Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (hdtp), other advanced touch user interfaces, and advanced mice

ABSTRACT

A hypermedia object associated with an application, displayed on a display screen, and responsive to information is disclosed. The information is provided by a user interface input device having two-dimensional pointing functions and at least one additional user-adjustable input for entering values from a range of more than two possible values. The hypermedia object includes a first visual representation of the hypermedia object for display in a first region of a display screen, an associated responsive area in a second region of the display for in activating the hypermedia object, and a procedure for allowing a user to activate the hypermedia object from a user-initiated action enacted on the user interface input device. Activating the hypermedia object enables the entry of at least one additional user-adjustable input value for use by the associated application.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/026,248, filed Feb. 12, 2011, which claims benefit of priority fromU.S. Provisional Application No. 61/303,898, filed Feb. 12, 2010, thecontents of all of which are hereby incorporated by reference herein intheir entireties.

COPYRIGHT & TRADEMARK NOTICES

A portion of the disclosure of this patent document may containmaterial, which is subject to copyright protection. Certain marksreferenced herein may be common law or registered trademarks of theapplicant, the assignee or third parties affiliated or unaffiliated withthe applicant or the assignee. Use of these marks is for providing anenabling disclosure by way of example and shall not be construed toexclusively limit the scope of the disclosed subject matter to materialassociated with such marks.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates to hypermedia objects such as hyperlink, rollover,button, slider, and menus in hypermedia documents, and user interfacesproviding simultaneously-adjustable interactively-controlled discreteand pseudocontinuous user-adjustable settings and parameters, and inparticular to adapting such hypermedia objects to accept and utilize theadditional number of simultaneously-adjustable interactively-controlleddiscrete and pseudocontinuous parameters, and uses in applications.

Overview of the Invention

The present invention provides various types of extensions to thetraditional and contemporary hypermedia objects such as hyperlink,roll-over, button, menu, and slider functions found in web browsers andhypermedia documents by leveraging additional and richer collections ofuser interface signals provided by, for example, a High DimensionalTouchpad (HTPD, as taught for example in U.S. Pat. No. 6,570,078 andpending U.S. patent application Ser. Nos. 11/761,978 and 12/418,605),Advanced Mice (for example as taught in U.S. Pat. No. 7,557,797 pendingU.S. patent application Ser. Nos. 12/619,678, 13/025,129, 13/024,569,and related pending U.S. patent applications), and other rich parameteruser interfaces (for example, popular advanced touch interfacesemploying multitouch and/or gestures). Additionally, images of the humanhand as captured by video cameras can be used as an enhancedmultiple-parameter interface responsive to hand positions and gestures(for example as taught in U.S. patent application Ser. No. 10/683,915).The collection of these various technologies will be collectivelyreferred to as Advanced Pointing Devices (“APD”). In a scaledbackimplementation, scrollwheel controls of a conventional computer mousecan be used to operate the extended features of the inventive hypermediaobjects such as hyperlink, roll-over, button, menu, and sliderfunctions.

The extensions provided by the invention include:

-   -   Directing additional user input into a hypermedia “hotspot” by        clicking on it in the case of a hyperlink; and    -   Directing additional user input into a hypermedia “hotspot”        simply from cursor overlay or proximity (i.e., without clicking        on it) in the case of a roll-over.        The resulting extensions will be called “Multiparameter        Hypermedia Objects” (“MHO”). Potential uses of the MHOs and more        generally extensions provided for by the invention include:    -   Using the additional user input to facilitate a rapid and/or        more detailed information gathering experience in an        easily-entered and completed “subsession” within a usage        session;    -   Potentially capturing notes from the sub-session for future use;    -   Potentially allowing the sub-session to retain state (such as        last image displayed);    -   Leaving the hypermedia “hotspot” without clicking out of it.        A number of user interface metaphors can be employed by the        invention and/or its use, including one or more of:    -   Creating a pop-up visual or other visual change responsive to        the rollover or hyperlink activation;    -   Rotating an object using rotation angle metaphors provided by        the APD;    -   Rotating a user-experience observational viewpoint using        rotation angle metaphors provided by the APD, for example, as        described in pending U.S. patent application Ser. No. 12/502,230        “Control of Computer Window Systems, Computer Applications, and        Web Applications via High Dimensional Touchpad User Interface”        by Seung Lim;    -   Navigating at least one (1-dimensional) menu, (2-dimensional)        pallet or hierarchical menu, or (3-dimensional) space.

Such extensions, features, and other aspects of the present inventionpermit far faster browsing, shopping, and information gleaning throughthe enhanced features of these extended functionality roll-over andhyperlink objects. The result is the advantageous employment of an APDto improve the throughput and ease of operation of a hypermediaapplication (for example, when doing online shopping, hypermediareference reviews, and surfing the web) so as to obtain more informationfar more quickly with far less effort.

SUMMARY OF THE INVENTION

For purposes of summarizing, certain aspects, advantages, and novelfeatures are described herein. Not all such advantages may be achievedin accordance with any one particular embodiment. Thus, the disclosedsubject matter may be embodied or carried out in a manner that achievesor optimizes one advantage or group of advantages without achieving alladvantages as may be taught or suggested herein.

In one aspect of the invention, various types of extensions to thetraditional and contemporary hypermedia objects (such as hyperlink,roll-over, button, menu, and slider functions) found in web browsers andhypermedia documents are extended to form extended hypermedia objects byleveraging additional and richer collections of user interface signalsprovided by, for example, a High Dimensional Touchpad (“HTPD,” as taughtin U.S. Pat. No. 6,570,078), Advanced Mice (as taught in U.S. Pat. No.7,557,797), other rich parameter user interfaces such as popular touchinterfaces employing multitouch and/or gestures, and images of the handcaptured by video cameras.

In another aspect of the invention, the inventive extended hypermediaobjects allow features such as interactive 3D rotations of depictedobjects, 3D immersion experiences in virtual venues, and a number othercapabilities of potentially high value to electronic commerce,education, documentation, and interactive entertainment.

In another aspect of the invention, image manipulation support forexamples of these new interactive capabilities is provided.

In another aspect of the invention a hypermedia object associated withan application for display on a display screen and responsive toinformation provided by a user interface input device comprisingtwo-dimensional pointing functions and at least one additionaluser-adjustable input for entering values from a range comprising morethan two values, the hypermedia object comprising:

-   -   a first visual representation of the hypermedia object on a        display screen, the first displayed visual representation for        display in a first region of the display associated with an        application;    -   an associated responsive area in a second region of the display,        the responsive area for use in activating the hypermedia object;    -   a procedure for activating the hypermedia object from a        user-initiated action enacted on a user interface input device;        wherein activation of the hypermedia object enables the entry of        at least one additional user-adjustable input value for use by        the associated application.

In another aspect of the invention, the first and second regions of thedisplay are the same region.

In another aspect of the invention, the hypermedia object furthercomprises a hyperlink function activated by user interface input devicewhen a cursor is positioned within the associated responsive area, thecursor position controlled by the two-dimensional pointing functions.

In another aspect of the invention, the hypermedia object comprises arollover function activated by using the user interface input device toposition the cursor within the associated responsive area, the cursorposition controlled by the two-dimensional pointing functions.

In another aspect of the invention, the hypermedia object comprises abutton function activated by the user interface input device when thecursor is positioned within the associated responsive area, the cursorposition controlled by the two-dimensional pointing functions.

In another aspect of the invention, the hypermedia object comprises aslider function.

In another aspect of the invention, the hypermedia object comprises amenu function.

In another aspect of the invention, the user input device is a computermouse comprising a first scrollwheel.

In another aspect of the invention, the user input device is a computermouse further comprising a second scrollwheel.

In another aspect of the invention, the user input device is a computermouse comprising a touchpad

In another aspect of the invention, the user input device is a computermouse comprising a High Definition Touch Pad (HDTP).

In another aspect of the invention, the user input device comprises atouch user interface responsive to gestures and the at least oneadditional user-adjustable input comprises at least one gesture.

In another aspect of the invention, the user input device comprises atouch user interface responsive to the yaw angle of a finger in contactwith the touch user interface and the at least one additionaluser-adjustable input is responsive to a measurement of the yaw angle.

In another aspect of the invention, the user input device comprises atouch user interface responsive to the roll angle of a finger in contactwith the touch user interface and the at least one additionaluser-adjustable input is responsive to a measurement of the roll angle.

In another aspect of the invention, the user input device comprises atouch user interface responsive to the pitch angle of a finger incontact with the touch user interface and the at least one additionaluser-adjustable input is responsive to a measurement of the pitch angle.

In another aspect of the invention, the user input device comprises atouch user interface responsive to at least two angles of a finger incontact with the touch user interface and the at least one additionaluser-adjustable input is responsive to a measurement of each of the twoangles

In another aspect of the invention, a second visual representation ofthe hypermedia object is displayed when the hypermedia object isactivated.

In another aspect of the invention, the first visual representation ofthe hypermedia object changes when the hypermedia object is activated.

In another aspect of the invention, the first displayed visualrepresentation of the hypermedia object changes responsive to the atleast one additional user-adjustable input.

In another aspect of the invention, the user input device is a touchinterface comprising a tactile grammar.

In another aspect of the invention is used to implement an improvedinteractive interface for consumers viewing products online providingusers with a virtual, 3D view of the product as seen in stores,utilizing the HDTP to spatially manipulate views of the product as ifhandling and rotating the product.

In another aspect of the invention, as the at least one additionaluser-adjustable input for entering values is varied between values, oneor another image of a group of images is displayed as part or all of thedisplayed visual appearance of an MHO, directly responsive to the lastreceived value of the at least one additional user-adjustable input.

In another aspect of the invention, as the at least one additionaluser-adjustable input for entering values is varied between values, oneor another image of a group of images is displayed as part or all of thedisplayed visual appearance of an MHO, responsive to commands or dataprovided by an associated program and/or other software.

In another aspect of the invention is used to implement an improvedinteractive interface for consumers buying event tickets onlineproviding users with a virtual view of the event venue as seen from anyof the seats available for purchase. Those who have never been to agiven venue can experience the view from a given seat to inform theirselection before a purchasing decision is made.

In another aspect of the invention, image processing is used tosynthesize an image of a particular viewing angle from one or morephotographic images comprising one or more other viewing angle(s), atleast one of the calculation and display of which is under the controlof a user input device.

In another aspect of the invention, a mathematical model tied to ascaled seat-map or a database linked to a seat map can calculate and/orretrieve separation distance data and viewing angle data and present toone or both of an at least one image selection element and an at leastone image processing element, at least one of the calculation anddisplay of which is under the control of a user input device.

In another aspect of the invention, separation distance data and viewingangle data is used by at least one image processing element to calculatea synthesized view from one or more photographic images, at least one ofthe calculation and display of which is under the control of a userinput device.

In another aspect of the invention, an image selection element selectsimages to display based on calculated and/or retrieved viewing angledata. In an embodiment, an image selection element selects images todisplay based on calculated and/or retrieved separation distance data.

In another aspect of the invention, should some locations in the venuecontain view obstructions, the image selection element can includeprovisions for selection specific image selection fromobstruction-handling families of images, at least one of the calculationand display of which is under the control of a user input device.

In another aspect of the invention, distance and/or angle informationcan be used by an image processing element to provide one or more ofselective cropping and/or distance-varying image warping to render areasonably accurate expected view from the particular location, such asa seat or region of seats in a theater, sports, or performance venue, atleast one of the calculation and display of which is under the controlof a user input device.

In another aspect of the invention, at least some of the above can beused to provide display of images representing interactively selectedviews at various angles at the particular seat or seating area, at leastone of the calculation and display of which is under the control of auser input device.

In another aspect of the invention, these and additional imageprocessing functions can be used to implement panoramically mergedimages, at least one of the calculation and display of which is underthe control of a user input device.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments, taken in conjunction with theaccompanying drawing figures.

FIG. 1 depicts a plurality of windows, one or more of which can be ahypermedia application window such as a browser, and hierarchies ofvisually displayed and other objects within or associated with thesewindows.

FIG. 2 depicts typical operation of a traditional hyperlink.

FIG. 3 depicts modes of navigation among web pages or hypermedia views.

FIG. 4 depicts a rectangular region of a window or hypermedia object ina window, defining quantities and geometric aspects relevant toconditional tests for cursor location within a displayed area.

FIG. 5 depicts a pair of button objects and actions resulting from theiroperation.

FIG. 6 depicts a rollover object and actions resulting from itsoperation.

FIG. 7 depicts a menu object and actions resulting from its operation.

FIG. 8 depicts a pair of slider objects and actions resulting from theiroperation.

FIG. 9 provides a summary table of example traditional hyperobjects.

FIG. 10 illustrates the side view of a finger lightly touching thesurface of a tactile sensor array.

FIG. 11 depicts a popularly accepted model of a typical cell phone orPDA capacitive proximity sensor implementation.

FIG. 12a is a graphical representation of a tactile image produced bycontact of a human finger on a tactile sensor array. FIG. 12b provides agraphical representation of a tactile image produced by contact withmultiple human fingers on a tactile sensor array of lesser spatialresolution than that depicted in FIG. 12 a.

FIG. 13 depicts a signal flow in an HDTP implementation.

FIGS. 14a-14f illustrate the six independently adjustable degrees offreedom of touch from a single finger that can be simultaneouslymeasured by the HDTP technology.

FIG. 15 suggests general ways in which two or more of theseindependently adjustable degrees of freedom adjusted simultaneously.

FIG. 16 demonstrates a few two-finger multi-touch postures and/orgestures from the many that can be readily recognized by HDTPtechnology.

FIG. 17 shows an example of how raw measurements of the six quantitiesof FIGS. 14a-14f , together with shape recognition for distinguishingcontact with various parts of the hand and the touchpad, can be used tocreate a rich information flux of parameters, rates, and symbols.

FIG. 18 shows an approach for incorporating posture recognition, gesturerecognition, and other functions to create a rich human/machine tactileinterface system capable of additionally supporting or incorporatingsyntax and grammars.

FIGS. 19a-19d depict operations acting on various parameters, rates, andsymbols to produce other parameters, rates, and symbols, includingoperations such as sample/hold, interpretation, context, etc.

FIG. 20 depicts a user interface input arrangement incorporating one ormore HDTPs that provides user interface input event and quantity routingfor focus control.

FIGS. 21a-21g depict a number of arrangements and embodiments employingHDTP technology.

FIGS. 22a-22e depict various integrations of an HDTP into the back of aconventional computer mouse as taught in U.S. Pat. No. 7,557,797.

FIGS. 23a and 23b illustrate examples of conventional scroll-wheel mouseprovided with an added left-right scroll-wheel as taught in U.S. Pat.No. 7,557,797.

FIGS. 24a-24c illustrate examples where a single trackball isincorporated into the back of a conventional computer mouse as taught inU.S. Pat. No. 7,557,797.

FIGS. 25a-25c illustrate examples where two trackballs are incorporatedinto the back of a conventional computer mouse as taught in U.S. Pat.No. 7,557,797, some of these (FIGS. 25b-25c ) comprising yet otheradditional sensors.

FIG. 25d depicts a mouse provided with a trackball and a small touchpadas taught in U.S. Pat. No. 7,557,797.

FIG. 25e depicts a mouse provided with a plurality of slider controls astaught in U.S. Pat. No. 7,557,797.

FIGS. 26a-26c depicts exemplary embodiments providing HDTP technologieswith a HID device abstraction for interfacing to applications.

FIGS. 27a-27d depict arrangements for directing additional userinterface parameter signals to applications.

FIG. 28a depicts a browser plug-in arrangement for directing additionaluser interface input parameters to an application interfacing to thebrowser.

FIG. 28b depicts additional browser plug-in arrangements for advanced 2Dand/or 3D vector and raster graphics which can additionally beinterfaced with client-side software such as JAVA Script.

FIGS. 29a and 29b depict arrangements provided for by the inventionwherein an application designed to utilize additional APD user interfaceparameter signals.

FIGS. 30a and 30b depict arrangements for directing additional userinterface input parameters to multi-parameter hypermedia objects (MHOs)rendered within the display of browser-based applications.

FIGS. 31-34 depict MHOs that differ from direct extensions oftraditional and contemporary hypermedia objects.

FIG. 35 depicts a 1-dimensional array of N images, any one of which canbe selected for rendering as part or all of the displayed visualappearance of an MHO.

FIG. 36 depicts a periodic structure imposed on the array of FIG. 34.

FIG. 37 depicts a 2-dimensional array of N×M images, any one of whichcan be selected for rendering as part or all of the displayed visualappearance of an MHO.

FIG. 38a depicts a 1-dimensional periodic structure imposed on the2-dimensional array of FIG. 36.

FIG. 38b depicts a 2-dimensional periodic structure imposed on the2-dimensional array of FIG. 37.

FIG. 39 depicts a 3-dimensional array of K×M×N images, any one of whichcan be selected for rendering as part or all of the displayed visualappearance of an MHO.

FIG. 40 depicts a 1-dimensional periodic structure imposed on the3-dimensional array of FIG. 39.

FIG. 41 depicts a body page that comprises various types of traditionalhypermedia objects and MHOs.

FIG. 42 shows a specific section of the webpage of FIG. 41 so as tofocus on MHO features.

FIG. 43 provides a showcase view of a single portion of the webpagesection of FIG. 42.

FIG. 44 shows the result of selecting a displayed element shown in FIG.43 after choosing a color from the list shown in FIG. 43.

FIGS. 45a-45f depict views of a shoe as can be captured in individualimages for use in the image array data structures described inconjunction with FIGS. 34-40.

FIG. 46a depicts a sequence of images capturing a 1-dimensional rotationof a shoe over a non-periodic range of rotation angles as controlled byyaw positions and movements of a single finger in contact with an HDTP.

FIG. 46b depicts a sequence of images capturing a 1-dimensional rotationof a shoe over a periodic range of rotation angles as controlled by yawpositions and movements of a single finger in contact with an HDTP.

FIG. 47 depicts a sequence of images capturing a 1-dimensional rotationof a shoe over a periodic range of rotation angles as controlled by afamily of two-finger posture positions and movements finger in contactwith an HDTP.

FIG. 48 depicts a sequence of images capturing a 1-dimensional rotationof a shoe within a different angle of rotation that that of FIGS.46a-46b over a range of rotation angles as controlled by the pitch angleof a single finger in contact with an HDTP.

FIG. 49 depicts an event venue ticket shopping webpage.

FIG. 50 depicts a popup image window overlay on the ticket shoppingwebpage depicted in FIG. 49, said popup image depicting a view showing aphotographic representation of the view of a stage.

FIG. 51 depicts a popup image window overlay on the ticket shoppingwebpage depicted in FIG. 49, said popup image depicting an overhead viewshowing a photographic representation of the separation distance from astage.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments may be utilized, and structural, electrical, as wellas procedural changes may be made without departing from the scope ofthe present invention.

In the following description, numerous specific details are set forth toprovide a thorough description of various embodiments. Certainembodiments may be practiced without these specific details or with somevariations in detail. In some instances, certain features are describedin less detail so as not to obscure other aspects. The level of detailassociated with each of the elements or features should not be construedto qualify the novelty or importance of one feature over the others.

Windowing Systems

Desktop, laptop, tablet, web, and other types of contemporary computersprovide for a plurality of active software applications to share visualdisplay and user input devices by means of some form of windowingsystem. Windowing systems are well known with foundational principlesdating back decades (see for example F. R. Hopgood, et al., Methodologyof Window Management, Springer-Verlag, Berlin, 1986, ISBN 0387161163)and are known at least as an operational level to virtually all users ofthese devices. Without getting into the many well-known aspects ofwindowing systems, one skilled in the art is reminded that:

-   -   A plurality of windows can be displayed simultaneously on the        screen of the (desktop, laptop, tablet, or web) computer;    -   Multiple windows can overlap one another;    -   The windowing system also provides a visually-rendered cursor        whose position is determined by left-right/forward-back        operation provisions of a pointing device (mouse, touchpad,        trackball, etc.);    -   Windows are typically selected by “clicking” a discrete-event        provision (button operation, touchpad tap, etc.) of the pointing        device—windows can also be selected by default in some cases,        such as when the initialization of a previously inactive        application displays, updates, or popsup a new window;    -   A selected window remains selected until the user selects a        different window or a window is selected by default;    -   User keyboard input and other types of pointing device input is        typically directed to aspects of an application associated with        the window that is currently selected.

FIG. 1 depicts a visual display screen area 100 displaying a pluralityof representative windows—here 101, 102, 110. In this FIG., none of thewindows 101, 102, 110 are shown as overlapping so as to streamline thediscussion; one skilled in the art will understand the appearance ofoverlapping of one or more of these windows. One or more of thesewindows can be a hypermedia window such as a browser (here 110), andhierarchies of objects (111 and 111.1; 112 and 112.1, 112.2) renderedwithin or superimposed over the display area of the browser window 110.The hypermedia (browser or application) window 110 of FIG. 1 alsodepicts a toolbar 110.tb as well as a vertical scrollbar 110.vs and ahorizontal scrollbar 110.hs. Such vertical and horizontal scrollbarstypically appear when the display area within the window is smaller thanthe vertical and/or horizontal span of the visual content, allowingcontrol of the vignette displayed within the aperture created by thedisplay, this control responsive to the positions of scrollbar(s) withinthe degrees of possible travel. The position of a scrollbar is in turncontrolled by one or more input aspects from the pointing device.

Browser and Other Hypermedia Windows in a Windowing Environment

As is also well-known, a particularly important type of window-basedapplication is the browser. The browser visually renders informationaccording to the directives of one or more of a markup language file(HTML, XML, etc.), a script (such as JAVA script, etc.), and/or othersource material (for example, from a PDF viewer). The browser can bedriven from one or more files and/or executing program(s) that are onthe computer itself and/or at one or more server computers reachableover an attached network. The attached network can be a closed localarea network, but far more typically the attached network providesaccess to the internet. Most usage of browsers is to access files and/orexecuting program(s) that are hosted by one or more servers reachablevia the internet.

Navigation Via Hyperlinks and Page Arrow Buttons

Browsers and some other type of applications can include various typesof visually-cued ‘hyperlink’ objects that can be spatially-localizedwithin the window (underlined, highlighted, and/or colored text, imagesor regions within images, etc.) that can be operated by pointing deviceinput events such as clicking when the screen cursor is overlaid orsufficiently close to the spatial location of the visual cue. Operationof the hyperlink typically either changes the material displayed in thewindow or associated additional window, or causes the display of a newwindow as is depicted in FIG. 2.

Because such windows typically display combinations of text, graphics,images, and/or video, portions or entireties of which can implementhyperlinks, browsers and similarly-featured application windows can bereferred to as hypermedia application windows.

FIG. 3 depicts an example of navigation among web pages in a browserwindow or views within another type of hypermedia application window.Hyperlinks themselves provide stepwise means for moving forward in asequence of views, while toolbar 110.tb page arrow buttons, such as thebuttons 110.fwd and 110.bwd, operated by input from the pointing device,allows one to move forward 311 or backward 312 within the sequence ofweb pages or views 301-304 traversed up to the time of their operation.It is noted that, in typical implementations, branching into analternative hyperlink from a previous web page or view (any of 301, 302,303) destroys all forward history (respectively 302-304, 303-304, 304).Some contemporary tabbed browsers overcome this limitation by creating anew tab that inherits the actively displayed webpage/view and history.

Contemporary Hyperlink, Rollover, Button, Slider, and Menu Objects

In this section, traditional and contemporary hypermedia objects such ashyperlink objects, button objects, rollover objects, menu objects, andslider objects (including scrollbar sliders and zoom sliders which areadditionally often controlled by the scrollwheel of a scrollwheel mousewhenever a window or frame within a window is selected) are consideredand compared.

Cursor Contac/Proximity Detection

Each of the traditional hyperlink, rollover, button, slider, and menuobjects described below require conditional tests on the location of thecursor. As to this, FIG. 4 depicts a representative rectangular regionof a window or hypermedia object in a window. In this example, thepresence of the cursor in the window can be determined by a conditiontest made on the coordinates of the cursors position, for example withrespect to the situation depicted in FIG. 4 such a conditional test canbe structured as:

-   -   If [(500≦x≦800) && (300≦y≦700)], then [“Active Action”], else        [“Inactive Action”];

Note that when both the “x” and “y” coordinates are within the rangedefined by the conditional test, contact with the window is subsequentlydefined.

This form of conditional test thus comprises effectively the entirety ofwhat is required to operate the rollover object and to pre-activate ordeactivate a menu items in the aforementioned example menu object. Notethat when both the “x” and “y” coordinates are within the range definedby the conditional test, contact with a “hotspot” comprising thegeometric scope of the hyperobject is subsequently defined.

The hyperlink, button, menu item, and some slider operation modesrequire additional inclusion of a click event:

-   -   if [(500≦x≦800) && (300≦y≦700) && (ClickEvent”=TRUE)],        -   then [“Active Action”];        -   else [“Inactive Action”];

The same menu item and slider operation modes require additionalinclusion of a click-and-drag event which combines the above withadditional conditional tests.

Hyperlink Object

By long-accepted definition and convention, the traditional hyperlinkobject described requires the cursor to be located on or sufficientlynear the visually rendered hyperlink element and a subsequent clickevent to activate. Typically the traditional hyperlink object alsoprovides some visible responsive indication of its operation by thepointer device. Activation of the traditional hyperlink can change thecurrently displayed webpage or view to a new webpage or view, launch apop-up window or a new window directed to a new webpage or view, launcha pop-up menu, or provide other operations.

Button Object

A traditional hyperobject related to the hyperlink is the button object.The button is very similar in its operation to a graphical iconhyperlink, and can be used in the same manner, but in that it istypically graphically rendered to resemble a physically-operated“control panel” pushbutton it is most often employed in panel controlfunctions such as an “Enter” function, data entry functions, modecontrol functions, start/stop/pause functions, etc. FIG. 5 depicts apair of representative button objects and actions resulting from theiroperation.

Rollover Object

Another traditional hyperobject arguably related to the hyperlink is therollover object. The rollover object typically offers the same range ofactions as a button or hyperlink but only requires the cursor to belocated on or sufficiently near the visually rendered hyperlink elementto activate (i.e., no subsequent click event is used). Because therollover object is so readily activated by causal movement of thecursor, the rollover object typically performs rapidly reversibleoperations such as simply changing its visual representation (forexample, changing from one image or graphic to another image orgraphic). In some cases, a rollover function and a hyperlink functioncan be combined so as to allow a rollover event to invoke a rapidlyreversible operation and a click event to invoke a more action resultingin more significant actions (such as changing the currently displayedwebpage or view to a new webpage or view, launch a pop-up window or anew window directed to a new webpage or view, etc.). FIG. 6 depicts arepresentative rollover object and actions resulting from its operation.

Menu Object

Yet another traditional hyperobject related to the hyperlink, button,and rollover is the menu object. In practice there are a wide range oftypes of menus and operational procedures for them; here only arepresentative example is considered. The menu usually pops up as aresult of the operation of a button or rollover object. Once the menu isdisplayed, moving the cursor over the individual menu items causes somesort of visual indication of pre-activation. A click caused apre-activated individual menu item to act as if it were a button. FIG. 7depicts a representative menu object of the type described in thisexample (mindful as is one skilled in the art that other types of menuoperations are known) and actions resulting from its operation.

Sliders

A somewhat more complicated traditional hyperobject is the slider,typically comprising a visually-displayed “knob” and a typically linearpath for which it is permitted to “travel,” said travel responsive toinput from a user input device. Although exceptions can be found and/orcontrived, a slider is usually employed to set or change the value of aparticular variable in an application program from within a rangecomprising many possible values. For example, a slider can be used as anaudio volume control, selecting from any of a number of possible volumevalues available, for example where the number of possible volume valuesavailable could typically be 16 to 128. In another application, theslider can be used to set the zoom-level on a map viewer. When thenumber of possible values available is small, the function provided bythe traditional slider object can alternatively be implemented by atraditional menu object. The value of the slider object stands forthwhen the number of possible values available is large (making operationof a traditional menu object unwieldy) or the metaphor and/or persistentvisual geometric indication of the selected value is desired. In atypical implementation, the slider button is clicked on and its positionis dragged in either direction along a one-dimensional travel path. Insome embodiments the mere change in the slider “knob” position issufficient to change the value of a selected variable from one value ina range comprising many possible values to another such value in therange. In other embodiments the click operation must be released inorder to change the value of a selected variable from one value in arange comprising many possible values to another such value. Many othertypes of slider implementations are apparent to one skilled in the art.FIG. 8 depicts a pair of representative slider objects and actionsresulting from their operation.

Noted extensions to the above characterization of sliders are typicallymade for scrollbar sliders (on edges of windows) and zoom sliders(employed in some applications such as map and image viewers). Forexample, once a window or frame of within a window that includes avertical scrollbar is selected, the scrollwheel provided by acontemporary computer mouse is solely directed to the operation of thevertical scrollbar slider (for example, the scrollbar 110.vs of FIG. 1)as long as that window or frame of within a window remains selected. Asanother example, once a window or frame of within a window that includesa zoom slider is selected, the scrollwheel provided by a contemporarycomputer mouse is solely directed to the operation of the zoom slider aslong as that window or frame of within a window remains selected. Manyvariations are possible for managing scrollwheel assignments when bothscrolling and zoom functions are available:

-   -   In some map and image viewer applications both scrollbar sliders        and a zoom slider are provided, for example versions of Google        Maps which provide a map image frame and a side-area frame,        wherein the scrollwheel controls the zoom slider when the image        frame is selected and the vertical scrollbar slider when the        side-area frame is selected.    -   In other map and image viewer applications only either scrollbar        sliders or a zoom slider is provided, wherein the scrollwheel is        directed to one of these while other provisions can be provide        for the other (for example, the scrollwheel is directed to the        zoom slider and click-and-drag actions on the image are used to        implement a scroll function in lieu of scrollbar sliders).    -   In other map and image viewer applications, neither scrollbar        sliders and zoom sliders are provided (for example, the        scrollwheel is directed to the zoom function even though no        slider is displayed, and click-and-drag actions on the image are        used to implement a scroll function in lieu of scrollbar        sliders).

Comparison

FIG. 9 provides a summary table of some of the attributes of the exampletraditional hyperobjects described and considered thus far. The examplesdescribed, and the further simplifications made in order to display inthe table, are only meant to be representative and are hardlycomprehensive as is clear to one skilled in the art.

Note in the traditional hyperobjects described and considered thus far,as well as the many variations and alternatives known by one skilled inthe art, virtually all rely on the following user-input driven from aconventional pointing device for their operation:

-   -   Location of the cursor;    -   Click/release events.

Traditional User Interface Pointing Devices

Turning now to traditional user interface pointing devices, thetraditional mouse, traditional trackball, and traditional touchpad, andtraditional touchscreen typically are used to provide the following userinputs:

Traditional User Input Traditional Traditional Touchpad or Type MouseTrackball Touchscreen Cursor “X” Left-right position Left-right rotationLeft-right position position of housing of trackball of finger/stylusCursor “Y” Forward-back Forward-back Left-right position positionposition of housing rotation of of finger/stylus trackball Left clickLeft button Left button Left button and/or tap Right click Right buttonRight button Right button Double (left) Double operation Doubleoperation Double operation click of Left button of Left button of Leftbutton and/or double-tap

Contemporary Generation User Interface Pointing Devices

More contemporary computer mice additionally provide a scrollwheel alongwith the traditional components and features of the traditional mouse.Some scrollwheels allow the wheel to be depressed downward to operate aspring-loaded switch that provides a third class of button events. Asmentioned just above, typically the scrollwheel provided by acontemporary computer mouse is solely directed to the operation of thevertical scrollbar (for example, the scrollbar 110.vs of FIG. 1), ifdisplayed, of the currently selected window. More recently, computermice providing “2-way scrolling” (sometimes called “4-way scrolling”)features wherein the scrollwheel. In addition to conventionalforward-back rotation, can be tilted left or right with the resultingsignal directed to the control the horizontal scrollbar (for example,the scrollbar 110.hs of FIG. 1), if displayed, of the currently selectedwindow.

Providing an additional scroll control to a scrollwheel mouse that canbe used to operate the horizontal scrollbar with a left-right operationwas taught several years prior to the appearance of such products in thespecification of issued U.S. Pat. No. 7,557,797 (priority date Feb. 12,2004) and is to be addressed in a pending continuation patentapplication from that specification subject to that same priority date.

Additionally, touch screens have recently received tremendous attentionwith the addition of array tactile imaging capabilities. Such touchscreen technologies permit multi-touch sensing, metaphors, and gestures.Although such touch screen technologies have obtained great commercialsuccess from there defining role in the iPhone and subsequentadaptations in PDAs and other types of cell phones and hand-helddevices, these were in fact taught in the 1999 filings of U.S. Pat. No.6,570,078 and pending U.S. patent application Ser. No. 11/761,978.

These more advanced user interface pointing devices provide additionaluser control capabilities that can be used in hypermedia applications,and in particular in web-based applications rendered in a browser. Aknown example of this is the aforementioned use of the scrollwheel incontrolling the degree of zoom in the web-based Google Maps application.

Further, there remains a wide range of additional control capabilitiesthat can be provided by further enhanced user interface technologies. Anumber of representative enhanced user interface technologies aredescribed next, specifically.

-   -   (a) the HDTP taught in the 1999 filings of U.S. Pat. No.        6,570,078 and pending U.S. patent application Ser. No.        11/761,978, pending U.S. patent application Ser. Nos.        12/418,605, 12/502,230, 12/541,948, and related pending U.S.        patent applications; and    -   (b) the Advanced Mice taught in the 2004 filings of issued U.S.        Pat. No. 7,557,797 and related pending U.S. patent applications        such as Ser. Nos. 12/619,678, 13/025,129, 13/024,569. The        capabilities of these, or to a more limited extent, the        capabilities of contemporary generation user interface pointing        devices can be used to enhance the capabilities of traditional        hypermedia objects (such as the hyperlink, button, rollover,        menu, and slider) as well as defining new types of hypermedia        objects.

HDTP User Interface Technology

In an embodiment, a touchpad used as a pointing and data entry devicecan comprise an array of sensors. The array of sensors is used to createa tactile image of a type associated with the type of sensor and methodof contact by the human hand. The tactile image comprises and array ofdata elements such as an array of pressure measurements, and array ofproximity measurements, an array of reflective optical measurements,etc. Thus the tactile image can be or comprise a pressure image,proximity image, reflective optical image, etc. In an embodiment, eachdata element comprises a scalar numerical value corresponding to ameasurement from an associated sensor. In another embodiment, at leastone data element comprises a plurality of scalar numerical values. In anembodiment, each data element comprises one or more scalar valuesproduced from signal processing, image processing, and/or otheroperations applied to measurements provided by an array of sensors.

In one embodiment, the individual sensors in the sensor array arepressure sensors and a direct pressure-sensing tactile image isgenerated by the sensor array.

In another embodiment, the individual sensors in the sensor array areproximity sensors and a direct proximity tactile image is generated bythe sensor array. Since the contacting surfaces of the finger or handtissue contacting a surface typically increasingly deforms as pressureis applied, the sensor array comprised of proximity sensors alsoprovides an indirect pressure sensing tactile image.

In another embodiment, the individual sensors in the sensor array can beoptical sensors. In one variation of this, an optical image is generatedand an indirect proximity tactile image is generated by the sensorarray. In another variation, the optical image can be observed through atransparent or translucent rigid material and, as the contactingsurfaces of the finger or hand tissue contacting a surface typicallyincreasingly deforms as pressure is applied, the optical sensor arrayalso provides an indirect pressure-sensing tactile image.

In some embodiments, the array of sensors can be transparent ortranslucent and can be provided with an underlying visual displayelement such as an alphanumeric and/or graphics and/or image display.The underlying visual display can comprise, for example, an LED arraydisplay, a backlit LCD, etc. Such an underlying display can be used torender geometric boundaries or labels for soft-key functionalityimplemented with the tactile sensor array, to display statusinformation, etc.

In an embodiment the touchpad can comprise a tactile sensor arrayobtains or provides individual measurements in every enabled cell in thesensor array that provides these as numerical values. The numericalvalues can be communicated in a numerical data array, as a sequentialdata stream, or in other ways. When regarded as a numerical data arraywith row and column ordering that can be associated with the geometriclayout of the individual cells of the sensor array, the numerical dataarray can be regarded as representing a tactile image.

The tactile sensor array should not be confused with the “null/contact”touchpad which, in normal operation, acts as a pair of orthogonallyresponsive potentiometers. These “null/contact” touchpads do not producepressure images, proximity images, or other image data but rather, innormal operation, two voltages linearly corresponding to the location ofa left-right edge and forward-back edge of a single area of contact.Such “null/contact” touchpads, which are universally found in existinglaptop computers, are discussed and differentiated from tactile sensorarrays in issued U.S. Pat. No. 6,570,078 and pending U.S. patentapplication U.S. Ser. No. 11/761,978 (pre-grant publication U.S.2007/0229477). Before leaving this topic, it is pointed out that thesethe “null/contact” touchpads nonetheless can be inexpensively adaptedwith simple analog electronics to provide at least primitive multi-touchcapabilities as taught in U.S. Pat. No. 6,570,078 and pending U.S.patent application U.S. Ser. No. 11/761,978 (therein, paragraphs[0022]-[0029] of its pre-grant publication U.S. 2007/0229477, forexample).

One implementation of a tactile sensor array is a pressure sensor array.Pressure sensor arrays discussed in U.S. Pat. No. 6,570,078 and pendingU.S. patent application Ser. No. 11/761,978. These typically operate bymeasuring changes in electrical (resistive, capacitive) or opticalproperties of an elastic material as the material is compressed.Prominent manufacturers and suppliers of pressure sensor arrays includeTekscan, Inc. (307 West First Street, South Boston, Mass., 02127,www.tekscan.com), Pressure Profile Systems (5757 Century Boulevard,Suite 600, Los Angeles, Calif. 90045, www.pressureprofile.com), SensorProducts, Inc. (300 Madison Avenue, Madison, N.J. 07940 USA,www.sensorprod.com), and Xsensor Technology Corporation (Suite 111,319-2nd Ave SW, Calgary, Alberta T2P OC5, Canada, www.xsensor.com).

In lieu of a pressure sensor array, a proximity sensor array oreffective equivalents (for example, as can be accomplished with a videocamera as described in issued U.S. Pat. No. 6,570,078 and pending U.S.patent application Ser. No. 11/761,978) can be used as a tactile sensorarray. In general, a tactile proximity sensor array suitable for usewith the present invention can be implemented in a wide variety of waysusing any number of techniques or physical effects. The only requirementis that the tactile proximity sensor array produce a multi-levelgradient measurement image as a finger, part of hand, or other pliableobject varies is proximity in the immediate area of the sensor surface.

More specifically, FIG. 10 illustrates a representative side view of afinger 1001 lightly touching the surface 1002 of a tactile sensor array.In this example, the finger 1001 contacts the tactile sensor surface ina relatively small area 1003. In this situation, on either side thefinger curves away from the region of contact 1003, where thenon-contacting yet pro xi mate portions of the finger grow increasinglyfar 1004 a, 1005 a, 1004 b, 1005 b from the surface of the sensor 1002.These variations in physical proximity of portions of the finger withrespect to the sensor surface should cause each sensor element in thetactile proximity sensor array to provide a corresponding proximitymeasurement varying responsively to the proximity, separation distance,etc. The tactile proximity sensor array advantageously comprises enoughspatial resolution to provide a plurality of sensors within the areaoccupied by the finger (for example, the area comprising width 1006). Inthis case, as the finger is pressed down, the region of contact 1003grows as the more and more of the pliable surface of the finger conformsto the tactile sensor array surface 1002, and the distances 1004 a, 1005a, 1004 b, 1005 b contract. If the finger is tilted, for example byrolling in the user viewpoint counterclockwise (which in the depictedend-of-finger viewpoint clockwise 1007 a) the separation distances onone side of the finger 1004 a, 1005 a will contract while the separationdistances on one side of the finger 1004 b, 1005 b will lengthen.Similarly if the finger is tilted, for example by rolling in the userviewpoint clockwise (which in the depicted end-of-finger viewpointcounterclockwise 1007 b) the separation distances on the side of thefinger 1004 b, 1005 b will contract while the separation distances onthe side of the finger 1004 a, 1005 a will lengthen.

Capacitive proximity sensors can be used in various handheld deviceswith touch interfaces (see for example, among many,http//electronics.howstuffworks.com/iphone2.htm,http://www.veritasetvisus.com/VVTP-12,%20Walker.pdf). Prominentmanufacturers and suppliers include Balda AG (Bergkirchener Str. 228,32549 Bad Oeynhausen, DE, www.balda.de), Cypress (198 Champion Ct., SanJose, Calif. 95134, www.cypress.com), and Synaptics (2381 Bering Dr.,San Jose, Calif. 95131, www.synaptics.com). In these sensors, the regionof finger contact is detected by variations in localized capacitanceresulting from capacitive proximity effects induced by a nearly-adjacentfinger. More specifically, the electrical field at the intersection oforthogonally-aligned conductive buses is influenced by the verticaldistance or gap between the surface of the sensor array and the skinsurface of the finger. The capacitive proximity sensor technology islow-cost, reliable, long-life, stable, and can readily be madetransparent. FIG. 11 (adapted fromhttp://www.veritasetvisus.com/VVTP-12,%20Walker.pdf with slightly morefunctional detail added) shows a popularly accepted model of a typicalcell phone or PDA capacitive proximity sensor implementation. In someembodiments the present invention can use the same spatial resolution ascurrent capacitive proximity touchscreen sensor arrays. In otherembodiments of the present invention, a higher spatial resolution isadvantageous. For example, in many contemporary capacitive proximitysensors, the touch of a fingertip can be comprised within the physicaldimensions of one sensor element or one sensor-separation spacing. Inhigher resolution implementations, the touch of a fingertip can span thephysical dimensions of many sensor elements and sensor-separationspacing, for example as in the higher resolution example depicted in(soon to be discussed) FIGS. 12a -12 b.

As a first example of an optical array sensor, Forrest M. Mims iscredited as showing that a conventional LED can be used as a lightdetector as well as a light emitter. Recently, light-emitting diodeshave been used as a tactile proximity sensor array (for example, asdepicted in the video available athttp://cs.nyu.edu/˜jhan/ledtouch/index.html). Such tactile proximityarray implementations typically need to be operated in a darkenedenvironment (as seen in the video in the above web link). In oneembodiment provided for by the invention, each LED in an array of LEDscan be used as a photodetector as well as a light emitter, although asingle LED can either transmit or receive information at one time. EachLED in the array can sequentially be selected to be set to be inreceiving mode while others adjacent to it are placed in light emittingmode. A particular LED in receiving mode can pick up reflected lightfrom the finger, provided by said neighboring illuminating-mode LEDs.The invention provides for additional systems and methods for notrequiring darkness in the user environment in order to operate an LEDarray as a tactile proximity sensor. In one embodiment, potentialinterference from ambient light in the surrounding user environment canbe limited by using an opaque pliable and/or elastically deformablesurface covering the LED array that is appropriately reflective(directionally, amorphously, etc. as can be advantageous in a particulardesign) on the side facing the LED array. Such a system and method canbe readily implemented in a wide variety of ways as is clear to oneskilled in the art. In another embodiment, potential interference fromambient light in the surrounding user environment can be limited byemploying amplitude, phase, or pulse width modulated circuitry and/orsoftware to control the underlying light emission and receiving process.For example, in an implementation the LED array can be configured toemit modulated light modulated at a particular carrier frequency orvariation waveform and respond to only modulated light signal componentsextracted from the received light signals comprising that same carrierfrequency or variation waveform. Such a system and method can be readilyimplemented in a wide variety of ways as is clear to one skilled in theart.

As a second example of an optical array sensor, use of video cameras forgathering control information from the human hand in various ways isdiscussed in U.S. Pat. No. 6,570,078 and Pending U.S. patent applicationSer. No. 11/761,978. In another video camera tactile controllerembodiment, a flat or curved translucent panel can be used as sensorsurface. When a finger is placed on the translucent panel, light appliedto the opposite side of the translucent panel reflects light in adistinctly different manner than in other regions where there is nofinger or other tactile contact. The image captured by an associatedvideo camera will provide gradient information responsive to the contactand proximity of the finger with respect to the surface of thetranslucent panel. For example, the parts of the finger that are incontact with the surface will provide the greatest degree of reflectionwhile parts of the finger that curve away from the surface of the sensorprovide less reflection of the light. Gradients of the reflected lightcaptured by the video camera can be arranged to produce a gradient imagethat appears similar to the multilevel quantized image captured by apressure sensor. By comparing changes in gradient, changes in theposition of the finger and pressure applied by the finger can bedetected.

In many various embodiments, the tactile sensor array can be connectedto interface hardware that sends numerical data responsive to tactileinformation captured by the tactile sensor array to a processor. Invarious embodiments, this processor will process the data captured bythe tactile sensor array and transform it various ways, for example intoa collection of simplified data, or into a sequence of tactile image“frames” (this sequence akin to a video stream), or into highly refinedinformation responsive to the position and movement of one or morefingers and/or other parts of the hand.

As to further representative detail of the latter example, a “frame” canrefer to a 2-dimensional list comprising a number of rows and a numberof columns forming an array, the array comprising tactile measurementvalue(s) for every sensor in a tactile sensor array at a given instance.In an embodiment, each data element comprises a scalar numerical valuecorresponding to a measurement from an associated sensor. In anotherembodiment, at least one data element comprises a plurality of scalarnumerical values. In an embodiment, each data element comprises one ormore scalar values produced from signal processing, image processing,and/or other operations applied to measurements provided by an array ofsensors. The time interval between one frame and the next one depends onthe frame rate of the system and the number of frames in a unit time(usually frames per second). FIG. 12a is a graphical representation of atactile image produced by contact with the bottom surface of the mostoutward section (between the end of the finger and the most nearbyjoint) of a human finger on a tactile sensor array. In this exampletactile array, there are 24 rows and 24 columns; other realizations canhave significantly more (hundreds or thousands) of rows and columns.Tactile measurement values of each cell are indicated by the numbers andshading in each cell. Darker cells represent cells with higher tactilemeasurement values. Similarly. FIG. 12b provides a graphicalrepresentation of an example tactile image produced by contact withmultiple human fingers on a tactile sensor array. In other embodiments,there can be a larger or smaller number of pixels for a given imagessize, resulting in varying resolution. Additionally, there can be largeror smaller area with respect to the image size resulting in a greater orlesser potential measurement area for the region of contact to belocated in or move about. (Note the sensor array of FIG. 11 has lessspatial resolution than that associated with FIG. 12b , which in turnhas less spatial resolution than that associated with FIG. 12 a.

Individual sensor elements in a tactile sensor array can varysensor-by-sensor when presented with the same stimulus. The inventionprovides for each sensor to be individually calibrated inimplementations where that can be advantageous. Sensor-by-sensormeasurement value scaling, offset, and/or nonlinear warpings can beinvoked for all or selected sensor elements during data acquisitionscans. Similarly, the invention provides for individual noisy ordefective sensors to be tagged for omission of their flawed measurementsduring data acquisition scans and/or post-scan data processing.

FIG. 13 depicts an example realization wherein a tactile sensor array isprovided with real-time or near-real-time data acquisition capabilities.The captured data reflects spatially distributed tactile measurements(such as pressure, proximity, etc.). The tactile sensory array and dataacquisition stage provides this real-time or near-real-time tactilemeasurement data to a specialized image processing arrangement for theproduction of parameters, rates of change of those parameters, andsymbols responsive to aspects of the hand's relationship with thetactile or other type of sensor array. In some applications, thesemeasurements can be used directly. In other situations, the real-time ornear-real-time derived parameters can be directed to mathematicalmappings (such as scaling, offset, and/or nonlinear warpings) inreal-time or near-real-time into real-time or near-real-timeapplication-specific parameters or other representations useful forapplications. In some embodiments, general purpose outputs can beassigned to variables defined or expected by the application.

FIGS. 14a-14f illustrate the six independently adjustable degrees offreedom of touch from a single finger that can be simultaneouslymeasured by the HDTP technology. The depiction in these figures is fromthe side of the touchpad. FIGS. 14a-14c show actions of positionalchange (amounting to applied pressure in the case of FIG. 14c ) whileFIGS. 14d-14f show actions of angular change. Each of these can be usedto control a user interface parameter, allowing the touch of a singlefingertip to control up to six simultaneously adjustable quantities inan interactive user interface. In more detail:

-   -   FIG. 14a depicts variation of the left/right position (“x”) of        the finger contact;    -   FIG. 14b depicts variation of the forward/back position (“y”) of        the finger contact;    -   FIG. 14c depicts variation of the up/down position or downward        pressure (“p”) of the finger contact;    -   FIG. 14d depicts variation of the clockwise/counterclockwise        (yaw) angle (“ψ”) of the finger contact;    -   FIG. 14e depicts variation of the left/right tilt (roll) angle        (“φ”) of the finger contact;    -   FIG. 14f depicts variation of the forward/back (pitch) angle        (“θ”) of the finger contact.

FIG. 15 suggests general ways in which two or more of theseindependently adjustable degrees of freedom adjusted at once with asingle finger 1500:

-   -   left/right position (“x”) of the finger contact 1511;    -   forward/back position (“y”) of the finger contact 1512;    -   up/down position or downward pressure (“p”) of the finger        contact 1516;    -   clockwise/counterclockwise (yaw) angle (“ψ”) of the finger        contact 1515;    -   left/right tilt (roll) angle (“φ”) of the finger contact 1513;    -   forward/back (pitch) angle (“θ”) of the finger contact 1514.

More advanced implementations of the HDTP provide for multitouchcapabilities that can be far more sophisticated that those popularizedby the Apple iPhone, NYU, and others.

FIG. 16 demonstrates a few representative two-finger multi-touchpostures and/or gestures from the hundreds that can be readilyrecognized by HDTP technology. HDTP technology can also be configured torecognize and measure postures and/or gestures involving three or morefingers, various parts of the hand, the entire hand, multiple hands,etc., as taught for example in U.S. Pat. No. 6,570,078 and pending U.S.patent application Ser. Nos. 11/761,978 and 12/418,605.

FIG. 17 shows an example of how raw measurements of the six quantitiesof FIGS. 14a-14f , together with shape recognition for distinguishingcontact with various parts of the hand and the touchpad, can be used tocreate a rich information flux of parameters, rates, and symbols, astaught for example in U.S. Pat. No. 6,570,078 and pending U.S. patentapplication Ser. Nos. 11/761,978 and 12/418,605.

FIG. 18 shows a representative approach for incorporating posturerecognition, gesture recognition, state machines, and parsers to createan even richer human/machine tactile interface system capable ofincorporating syntax and grammars, as taught for example in U.S. Pat.No. 6,570,078 and pending U.S. patent application Ser. Nos. 11/761,978and 12/418,605.

The HDTP affords and provides for yet further capabilities. For example,sequence of symbols can be directed to a state machine, as shown in FIG.19a , to produce other symbols that serve as interpretations of one ormore possible symbol sequences. In an embodiment, one or more symbolscan be designated the meaning of an “Enter” key, permitting for samplingone or more varying parameter, rate, and/or symbol values and holdingthe value(s) until, for example, another “Enter” event, thus producingsustained values as illustrated in FIG. 19b . In an embodiment, one ormore symbols can be designated as setting a context for interpretationor operation and thus control mapping and/or assignment operations onparameter, rate, and/or symbol values as shown in FIG. 19c . Theoperations associated with FIGS. 19a-19c can be combined to provide yetother capabilities. For example, the example arrangement of FIG. 19dshows mapping and/or assignment operations that feed an interpretationstate machine which in turn controls mapping and/or assignmentoperations. In implementations where context is involved, such as inarrangements such as those depicted in FIGS. 19b-19d , the inventionprovides for both context-oriented and context-free production ofparameter, rate, and symbol values. The parallel production ofcontext-oriented and context-free values can be useful to drive multipleapplications simultaneously, for data recording, diagnostics, userfeedback, and a wide range of other uses.

FIG. 20 depicts a representative user arrangement incorporating one ormore HDTP system(s) or subsystem(s) that provide(s) user interface inputevent and routing of HDTP produced parameter values, rate values,symbols, etc. to a variety of applications. In an embodiment, theseparameter values, rate values, symbols, etc. can be produced for exampleby utilizing one or more of the individual systems, individual methods,and/or individual signals described above in conjunction with thediscussion of FIGS. 17, 18, and 19 a-19 b. As discussed later, such anapproach can be used with other rich multiparameter user interfacedevices in place of the HDTP. An arrangement similar to that of FIG. 20is also taught in pending U.S. patent application Ser. No. 12/502,230“Control of Computer Window Systems, Computer Applications, and WebApplications via High Dimensional Touchpad User Interface” by Seung Lim,and FIG. 20 is adapted from FIG. 6e of that pending application (U.S.patent application Ser. No. 12/502,230) for further expansion here.Additional window manger input focus control for High DimensionalTouchpad (HDTP), Advanced Mice, and other multidimensional userinterfaces are taught in pending U.S. patent application Ser. No.13/026,097.

In an implementation approach or modality of operation for anarrangement such as the one of FIG. 20, the Focus Control element uses aselected subset of the information stream provided by the HDTP or otheruser interface device providing traditional user-adjustable inputssupplemented by additional user-adjustable inputs. The Focus Controlelement uses a selected subset of the information stream to interpretthe user's intention for the direction of focus among several windows,applications, etc. The figure shows only applications, but some of thesecan be replaced with application child windows, operating system,background window, etc. In this example, focus may be controlled by an{x,y} location threshold test and a “select” symbol event, althoughother information may be used in its place.

In an arrangement such as the one of FIG. 20, or in otherimplementations, at least two parameters are used for navigation of thecursor when the overall interactive user interface system is in a moderecognizing input from cursor control. These can be, for example, theleft-right (“x”) parameter and forward/back (“y”) parameter provided bythe touchpad. The arrangement of FIG. 20 includes a representativeimplementation of this.

Alternatively, these two cursor-control parameters can be provided byanother user interface device, for example another touchpad or aseparate or attached mouse (the latter to be discussed shortly in thecontext of FIGS. 22a-22e ).

In some situations, control of the cursor location can be implemented bymore complex means. One example of this is the control of location of a3D cursor wherein a third parameter must be employed to specify thedepth coordinate of the cursor location. For such situations, thearrangement of FIG. 20 would be modified to include a third parameter(for use in specifying this depth coordinate) in addition to theleft-right (“x”) parameter and forward/back (“y”) parameter describedearlier.

In an embodiment, focus control is used to interactively routing userinterface signals among applications. In most current systems, there isat least some modality wherein the focus is determined by either thecurrent cursor location or a previous cursor location when a selectionevent was made. In the user experience, this selection event typicallyinvolves the user interface providing an event symbol of some type (forexample a mouse click, mouse double-click touchpad tap, touchpaddouble-tap, etc). The representative arrangement of FIG. 20 includes animplementation wherein a select event generated by the touchpad systemis directed to the focus control element. The focus control element inthis arrangement in turn controls a focus selection element that directsall or some of the broader information stream from the HDTP system tothe currently selected application. (In FIG. 20, “Application K” hasbeen selected as indicated by the thick-lined box and information-flowarrows.)

In some embodiments, each application that is a candidate for focusselection provides a window displayed at least in part on the screen, orprovides a window that can be deiconified from an icon tray or retrievedfrom beneath other windows that may be obfuscating it. In someembodiments, if the background window is selected, focus selectionelement that directs all or some of the broader information stream fromthe HDTP system to the operating system, window system, and/or featuresof the background window. In some embodiments, the background window canbe in fact regarded as merely one of the applications shown in the rightportion of the arrangement of FIG. 20. In other embodiments, thebackground window can be in fact regarded as being separate from theapplications shown in the right portion of the arrangement of FIG. 20.In this case the routing of the broader information stream from the HDTPsystem to the operating system, window system, and/or features of thebackground window is not explicitly shown in FIG. 20.

Touchscreen and Other Embodiments of the HDTP

FIGS. 21a-21g and 22a-22e depict a number of representative arrangementsand embodiments employing the HDTP technology. FIG. 21a illustrates aHDTP as a peripheral that can be used with a desktop computer (shown) orlaptop) not shown). FIG. 21 b depicts an HDTP integrated into a laptopin place of the traditional touchpad pointing device. In FIGS. 21a-21bthe HDTP tactile sensor can be a stand-alone component or can beintegrated over a display so as to form a touchscreen. FIG. 21c depictsan HDTP integrated into a desktop computer display so as to form atouchscreen. FIG. 21d shows the HDTP integrated into a laptop computerdisplay so as to form a touchscreen.

FIG. 21e depicts an HDTP integrated into a cell phone, smartphone, PDA,or other hand-held consumer device. FIG. 21f shows an HDTP integratedinto a test instrument, portable service-tracking device, portableservice-entry device, field instrument, or other hand-held industrialdevice. In FIGS. 21e-21f the HDTP tactile sensor can be a stand-alonecomponent or can be integrated over a display so as to form atouchscreen.

FIG. 21g depicts an HDTP touchscreen configuration that can be used in atablet computer, wall-mount computer monitor, digital television, videoconferencing screen, kiosk, etc.

In at least the arrangements of FIGS. 21a, 21c, 21d, and 21g , or othersufficiently large tactile sensor implementation of the HDTP, more thanone hand can be used and individually recognized as such.

Embodiments incorporating the HDTP into a Traditional or ContemporaryGeneration Mouse

FIGS. 22a-22e depict various representative integrations of an HDTP intothe back of a conventional computer mouse. In FIGS. 22a-22d the HDTPtactile sensor can be a stand-alone component or can be integrated overa display so as to form a touchscreen. Such configurations have veryrecently become popularized by the product release of Apple “MagicMouse™” although such combinations of a mouse with a tactile sensorarray on its back responsive to multitouch and gestures were taughtearlier in pending U.S. patent application Ser. No. 12/619,678 (prioritydate Feb. 12, 2004) entitled “User Interface Mouse with TouchpadResponsive to Gestures and Multi-Touch.”

In another embodiment taught in the specification of issued U.S. Pat.No. 7,557,797 and associated pending continuation applications more thantwo touchpads can be included in the advance mouse embodiment, forexample as suggested in the arrangement of FIG. 22e . As with thearrangements of FIGS. 22a-22d , one or more of the plurality of HDTPtactile sensors or exposed sensor areas of arrangements such as that ofFIG. 22e can be integrated over a display so as to form a touchscreen.

Advanced Mice User Interface Technology

The HDTP in the above examples is used to supply more than thetraditional two user interface parameters provided by a conventionaluser interface input device such as a conventional computer mouse,trackball, touchpad, etc. The present invention provides for the use ofother user interface input arrangements and devices as alternatives toor in conjunction with one or more HDTPs. In this section the featuresand capabilities of Advanced Mice are briefly reviewed and set up fortheir use in embodiments of the invention. Focus control can beimplemented in a manner completely or nearly analogous with FIG. 20, aswell as other approaches (for example as will be presented later in thecontexts of later FIGS. 27a-27d and FIGS. 30a-30b ).

In a simple example, the scroll-wheel of a scroll-wheel mouse is used toprovide a third simultaneously adjustable user interface parameter. Inanother example, a second or yet more additional scroll-wheels can beadded to a conventional scroll-wheel mouse. The resultant collection ofscroll-wheels can be relatively positioned in parallel, oriented atorthogonal angles so as to support a coordinate-metaphor, positioned onthe sides of the mouse body, etc. FIGS. 23a and 23b illustrate examplesof conventional scroll-wheel mouse provided with an added left-rightscroll-wheel 2322 as taught in U.S. Pat. No. 7,557,797. Sucharrangements can employ a connecting cable, or the device can bewireless.

In another example of Advanced Mice, one or more trackballs can be addedto a conventional computer mouse. FIGS. 24a-24c illustrate exampleswhere a single trackball is incorporated into the back of a conventionalcomputer mouse as taught in U.S. Pat. No. 7,557,797. FIGS. 25a-25cillustrate examples where two trackballs are incorporated into the backof a conventional computer mouse as taught in U.S. Pat. No. 7,557,797.The trackballs in the arrangements of FIGS. 24a-24c and FIGS. 25a-25ccan be the conventional two degree of freedom type (roll left-right,roll away-towards) or can provide three to six degrees of freedom astaught in U.S. Pat. No. 7,557,797; U.S. patent application Ser. No.10/806,694. Such arrangements can employ a connecting cable, or thedevice can be wireless.

Another example Advanced Mice arrangement include thetrackball/touchpad/mouse combinations of FIGS. 25c and 25d and themultiple slider configuration of FIG. 25e , each taught in U.S. Pat. No.7,557,797. Other example Advanced Mice arrangements include those withtwo or more scroll wheels (for example as in pending U.S. patentapplication Ser. No. 13/024,569), a multiple-parameter joystickproviding three or more simultaneously adjustable user interface inputson the back of a mouse (for example as in pending U.S. patentapplication Ser. No. 13/025,129), and such a multiple-parameter joystickcombined with a trackball (for example as also in pending U.S. patentapplication Ser. No. 13/025,129).

Each of these arrangements can employ a connecting cable, or the devicecan be wireless.

Video Control

Additionally, images of the human hand as captured by video cameras canbe used as an enhanced multiple-parameter interface responsive to handpositions and gestures, for example as taught in pending U.S. patentapplication Ser. No. 10/683,915 and more specifically in paragraphs[314], [321]-[332], [411], [653], and (in view of paragraph [325]) alsoparagraphs [241]-[263] of that pending application's pre-grantpublication U.S. 2004/0118268.

Example Use of the Additional Parameters by Applications

The types of human-machine geometric interaction between the hand andthe HDTP facilitate many useful applications within a visualizationenvironment. A few of these include control of visualization observationviewpoint location, orientation of the visualization, and controllingfixed or selectable ensembles of one or more of viewing parameters,visualization rendering parameters, pre-visualization operationsparameters, data selection parameters, simulation control parameters,etc. As one example, the 6D orientation of a finger can be naturallyassociated with visualization observation viewpoint location andorientation, location and orientation of the visualization graphics,etc. As another example, the 6D orientation of a finger can be naturallyassociated with a vector field orientation for introducing syntheticmeasurements in a numerical simulation.

As yet another example, at least some aspects of the 6D orientation of afinger can be naturally associated with the orientation of a roboticallypositioned sensor providing actual measurement data. As another example,the 6D orientation of a finger can be naturally associated with anobject location and orientation in a numerical simulation. As anotherexample, the large number of interactive parameters can be abstractlyassociated with viewing parameters, visualization rendering parameters,pre-visualization operations parameters, data selection parameters,numeric simulation control parameters, etc.

In yet another example, the “x” and “y” parameters provided by the HDTPcan be used for focus selection and the remaining parameters can be usedto control parameters within a selected GUI.

In still another example, the “x” and “y” parameters provided by theHDTP can be regarded as a specifying a position within an underlyingbase plane and the roll and pitch angles can be regarded as a specifyinga position within a superimposed parallel plane. In a first exampleextension of the previous two-plane example, the yaw angle can beregarded as the rotational angle between the base and superimposedplanes. In a second example extension of the previous two-plane example,the finger pressure can be employed to determine the distance betweenthe base and superimposed planes. In a variation of the previoustwo-plane example, the base and superimposed plane cannot be fixed asparallel but rather intersect as an angle associated with the yaw angleof the finger. In the each of these, either or both of the two planescan represent an index or indexed data, a position, pair of parameters,etc. of a viewing aspect, visualization rendering aspect,pre-visualization operations, data selection, numeric simulationcontrol, etc.

USB HID Device Abstraction

The USB HID device class provides an open interface useful for bothtraditional computer pointing devices such as the standard computermouse and other user interface devices such as game controllers. The USBHID device class has also been used to interface with the Logitech3DConnexion SpaceNavigator™. The USB HID device class is currentlyspecified at the time of this patent application by at least the DeviceClass Definition for HID 1.11, currently available athttg://www.usb.org/develogers/devclass docs/HID111.pdf. More generally,the invention provides for the USB HID device class to be used for atleast additional user interface signals (user interface parameters)provided by the High Dimensional Touchpad (HDTP), Advanced Mice, andother multidimensional or rich parameter user interfaces that generateadditional user interface signals above those found in traditionalcomputer mice, touchpads, and trackballs. This can be done in a numberways, for example as taught in pending U.S. Patent Application61/435,401 and as described below in material adapted from that pendingUS patent application.

In a first exemplary embodiment, a USB HID device abstraction isemployed to connect a computer or other device with an HDTP sensor thatis connected to the computer via a USB interface. Here the exemplaryHDTP signal processing and HDTP gesture processing are implemented onthe computer or other device. The HDTP signal processing and HDTPgesture processing implementation can be realized via one or more of CPUsoftware, GPU software, embedded processor software or firmware, and/ora dedicated integrated circuit. FIG. 26a depicts an exemplaryimplementation of such an embodiment.

In another exemplary embodiment, a USB HID device abstraction isemployed to connect a computer or other device with an HDTP sensor andone or more associated processor(s) which in turn is/are connected tothe computer via a USB interface. Here the exemplary HDTP signalprocessing and HDTP gesture detection are implemented on the one or moreprocessor(s) associated with HDTP sensor. The HDTP signal processing andHDTP gesture processing implementation can be realized via one or moreof CPU software, GPU software, embedded processor software or firmware,and/or a dedicated integrated circuit. FIG. 26b depicts an exemplaryimplementation of such an embodiment.

In another exemplary embodiment, a USB HID device abstraction is used asa software interface even though no USB port is actually used. The HDTPsignal processing and HDTP gesture processing implementation can berealized via one or more of CPU software, GPU software, embeddedprocessor software or firmware, and/or a dedicated integrated circuit.FIG. 26c depicts an exemplary implementation of such an embodiment.Alternatively, ADPs can interface to a computer or other device in yetother ways. For example, a special purpose interface can be used. Asanother example, the Bluetooth networking standard can be used.

Support for Additional Parameters Via Existing or Extended WindowSystems and Operating Systems

The additional interactively-controlled parameters provided by HDTP(such as that taught in the 1999 filings of issued U.S. Pat. No.6,570,078 and pending U.S. patent application Ser. No. 11/761,978,pending U.S. patent application Ser. Nos. 12/418,605, 12/502,230,12/541,948, and related pending U.S. patent applications), Advanced Mice(such as that Mice taught in the 2004 filings of issued U.S. Pat. No.7,557,797 and related pending U.S. patent applications), and other richmultiparameter user interface devices supply moreinteractively-controlled parameters than the established numbersupported by conventional window and operating systems. Provisions tosupport the use of additional interactively-controlled parametersprovided by HDTP, Advanced Mice, and other rich multiparameter userinterface devices with existing or extended operating systems has beentaught in pending U.S. patent application Ser. No. 12/875,128. Somematerial from pending U.S. patent application Ser. No. 12/875,128 isdirectly adapted in this section for convenience. Additionally, imagesof the human hand as captured by video cameras can be used as anenhanced multiple-parameter interface responsive to hand positions andgestures, for example as taught in pending U.S. patent application Ser.No. 10/683,915 and more specifically in paragraphs [314], [321]-[332],[411], [653], and (in view of paragraph [325]) also paragraphs[241]-[263] of that pending application's pre-grant publication U.S.2004/0118268.

More generally, the invention provides for additional user interfaceparameter signals provided by the not only the High Dimensional Touchpad(HTPD) and Advanced Mice, but also other multidimensional or richparameter user interfaces providing additional user interface signalsabove those found in traditional computer mice, touchpads, andtrackballs. This fuller collection (HDTP, Advanced Mice, othermultidimensional or rich parameter user interface devices providingadditional user interface signals above those found in traditionalcomputer mice, touchpads, and trackballs) will be collectively referredto as Advanced Pointing Devices (APDs).

There is a number of ways that conventional window systems, windowmanagers, and operating systems can be used or adapted to support theadditional interactively-controlled parameters provided by an APD. A fewexamples are provided here, and other approaches are anticipated by theinvention.

In one approach, the entire (interactively-controlled) information fluxprovided by an APD is carried over the same framework used to carry thetraditional computer mouse/touchpad user interface signals fromconventional pointing devices. In one version of this approach, only thedriver for the APD need be added and recognized by the window systems,window managers, and operating systems. The window systems, windowmanagers, and operating systems then distributes the entire(interactively-controlled) information flux to the application selectedaccording to focus control implemented by the operating system. For somewindow systems, window managers, and operating systems, such an approachcan be implemented without modification. In other window systems, windowmanagers, and operating systems implementations, such an approach canrequire a modification to the window system, window manager, andoperating system. Should a particular existing window systems, windowmanagers, and operating systems resident on a computing device requiresuch modification, the invention provides for the modification to beimplemented via a downloadable patch or other form of update (forexample, using a data-storage media).

FIGS. 27a and 27b depict a representative rendering of this approach. Ineach figure, the driver for the APD presents traditional computermouse/touchpad user interface signals from conventional pointing devices(thin straight arrowed lines) to the window systems, window managers,and operating systems as well as additional computermouse/touchpad/trackball user interface signals (thick straight arrowedlines) from the APD. In each of these approaches, as well as othervariations clear to one skilled in the art, the window system, windowmanager, and operating system, or combination of these comprise a focuscontrol functionality used to route the traditional and additional userinterface parameter signals. The focus control can be responsive to atleast the position of a displayed cursor with respect to a displayedapplication window, the cursor and application window displayed on adisplay screen. In some approaches or operating modes, merelypositioning the cursor within the window of an application makes a focusselection to that application. In other approaches or operating modes,positioning the cursor within the window of an application is not alonesufficient to make a focus selection to that application; instead thefocus stays with the last selection until a user-provided selectionevent is made, for example a mouse click or double click, a touchpad tapor double-tap, a trackball button click or double click, etc.

In the rendering of FIGS. 27a and 27b , focus control (for example, asdefined by cursor location with respect to one or more displayedapplication windows) is responsive traditional computer mouse/touchpaduser interface signals (thin straight arrowed lines). In otherarrangements, such as a system employing a 3D desktop, at least oneadditional parameter can be also directed to focus control and/or cursorlocation. In the suggestive rendering of FIGS. 27a and 27b , there are aplurality of applications, some designed to accept only traditionalcomputer mouse/touchpad user interface signals (in the upper right ofeach figure) as well as other applications designed to accept thesetraditional signals as well as one or more of the additional userinterface signals provided by the APD (in the lower right of eachfigure). The applicable portions of the description applies even ifthere are fewer applications of either or both types, or if there isonly one type or only one application overall. In the case of FIG. 27a ,the focus control routes only the traditional interface signals to aselected application designed to accept only traditional computermouse/touchpad user interface signals. In the case of FIG. 27b , thefocus control routes a larger collection of signals, including bothtraditional computer mouse/touchpad user interface signals as well as atleast one additional user interface signal made available by the APD.

In another approach, the windowing and/or operating system onlydistributes traditional computer mouse/touchpad user interface signalsfrom conventional pointing devices and other provisions are used todirect the additional user interface parameter signals provided by theAPD to selected applications. This can be implemented in a number ofways. In one example, depicted in FIG. 27c , separate focus controls areused, each responsive to the traditional user interface signals providedby the APD. In another example, depicted in FIG. 27d , the operatingsystem focus control provides signals to the routing element for theadditional user interface parameter signals provided by the APD. Othervariations are anticipated and are provided for by the invention.

Once user interface signals are routed to an application, theapplication it self can utilize or sub-route the user interface signalsin various ways. Some applications, such as data visualization, maps,simulations, CAD systems, etc. can beneficially use more than threesimultaneously interactively adjustable user inputs directly. Otherapplications, such as browsers and viewers, can support suchapplications indirectly as taught and discussed for example in pendingU.S. patent application Ser. No. 12/875,119. Browsers, viewers, andhypermedia documents can also be provided with advanced hypermediaobjects that generalize the notion of hyperlinks, rollovers, sliders,buttons, etc. that are configured to utilize additional user interfacesignals; such advanced hypermedia objects taught and discussed forexample in pending U.S. Patent Application 61/435,395.

Support for Additional Parameters Via Browser Plug-Ins

The additional interactively-controlled parameters provided by the APDprovide more than the usual number supported by conventional browsersystems and browser networking environments.

The use of browser plug-ins to support the use of HDTP (such as taughtin the 1999 filings of issued U.S. Pat. No. 6,570,078 and pending U.S.patent application Ser. No. 11/761,978, pending U.S. patent applicationSer. Nos. 12/418,605, 12/502,230, 12/541,948, and related pending U.S.patent applications), Advanced Mice (such as those taught in the 2004filings of issued U.S. Pat. No. 7,557,797 and related pending U.S.patent applications), and other rich multiparameter user interfacedevices with associated browser-based applications has been taught inpending U.S. Patent Application 61/239,428. Some of that material frompending U.S. Patent Application 61/239,428 is directly adapted in thissection for convenience. Additionally, images of the human hand ascaptured by video cameras can be used as an enhanced multiple-parameterinterface responsive to hand positions and gestures, for example astaught in pending U.S. patent application Ser. No. 10/683,915 and morespecifically in paragraphs [314], [321]-[332], [411], [653], and (inview of paragraph [325]) also paragraphs [241]-[263] of that pendingapplication's pre-grant publication U.S. 2004/0118268.

In an additional approach, the invention provides for an APD (whichagain includes the HDTP, Advanced Mice, and other rich or multiparameteruser interface devices) to interface with a browser via a browserplug-in. Such an arrangement can be used to capture the additional userinterface input parameters and pass these on to an applicationinterfacing to the browser. An example of such an arrangement isdepicted in FIG. 28a . The browser can interface with local or web-basedapplications that drive the visualization and/or control the datasource(s), process the data, etc. The browser can be provided withclient-side software such as JAVA Script.

The invention further provides for advanced graphics to be rendered in abrowser. This allows for implementations wherein the browser is used asa viewer for data visualizations, advanced animations, etc. The browsercan interface with local or web-based applications that drive thevisualization. An example arrangement is depicted in FIG. 28b . In anembodiment, the browser can be provided with Simple Vector Graphics(“SVG”) utilities (natively or via an SVG plug-in) so as to render basic2D vector and raster graphics. In another embodiment, the browser can beprovided with a 3D graphics capability, for example via the Cortona 3Dbrowser plug-in. These embodiments or alternatives can be provided withclient-side software such as JAVA Seri pt.

Multiple Parameter Extensions to Traditional Hypermedia Objects

The present invention provides extensions to the traditional andcontemporary hyperlink, roll-over, button, menu, and slider functionsfound in web browsers and hypermedia documents leveraging additionaluser interface parameter signals provided by an APD (i.e., HDTP,Advanced Mouse, or other rich or multiparameter user interfacesincluding currently popular advanced touch interfaces employingmultitouch and/or gestures). The extensions provided by the inventioninclude:

-   -   In the case of a hyperlink, button, slider and some menu        features, directing additional user input into a hypermedia        “hotspot” by clicking on it;    -   In the case of a roll-over and other menu features: directing        additional user input into a hypermedia “hotspot” simply from        cursor overlay or proximity (i.e., without clicking on it).        The resulting extensions will be called “Multiparameter        Hypermedia Objects” (“MHO”).

Potential uses of the MHOs and more generally extensions provided for bythe invention include:

-   -   Using the additional user input to facilitate a rapid and/or        more detailed information gathering experience in a low-barrier        sub-session;    -   Potentially capturing notes from the sub-session for future use;    -   Potentially allowing the sub-session to retain state (such as        last image displayed);    -   Leaving the hypermedia “hotspot” without clicking out of it.

A number of user interface metaphors can be employed in the inventionand/or its use, including one or more of:

-   -   Creating a pop-up visual or other visual change responsive to        the rollover or hyperlink activation;    -   Rotating an object using rotation angle metaphors provided by        the APD;    -   Rotating a user-experience observational viewpoint using        rotation angle metaphors provided by the APD, for example, as        described in pending U.S. patent application Ser. No. 12/502,230        “Control of Computer Window Systems, Computer Applications, and        Web Applications via High Dimensional Touchpad User Interface”        by Seung Lim;    -   Navigating at least one (1-dimensional) menu, (2-dimensional)        pallet or hierarchical menu, or (3-dimensional) space.

In an embodiment, a second displayed visual representation of thehypermedia object is displayed on the display screen when the hypermediaobject is activated.

In an embodiment, the first displayed visual representation of thehypermedia object changes when the hypermedia object is activated.

In an embodiment, the first displayed visual representation of thehypermedia object changes is responsive to the at least one additionaluser-adjustable input.

In an embodiment, a hypermedia object associated with an application fordisplay on a display screen and responsive to information provided by auser interface input device comprising two-dimensional pointingfunctions and at least one additional user-adjustable input for enteringvalues from a range comprising more than two values, the hypermediaobject comprising:

-   -   a first visual representation of the hypermedia object on a        display screen, the first displayed visual representation for        display in a first region of the display associated with an        application;    -   an associated responsive area in a second region of the display,        the responsive area for use in activating the hypermedia object;    -   a procedure for activating the hypermedia object from a        user-initiated action enacted on a user interface input device;        wherein activation of the hypermedia object enables the entry of        at least one additional user-adjustable input value for use by        the associated application.

In an embodiment, the first and second regions of the display are thesame region.

In an embodiment, the hypermedia object further comprises a hyperlinkfunction activated by the user interface input device when a cursor ispositioned within the associated responsive area, the cursor positioncontrolled by the two-dimensional pointing functions.

In an embodiment, the hypermedia object comprises a rollover functionactivated by using the user interface input device to position thecursor within the associated responsive area, the cursor positioncontrolled by the two-dimensional pointing functions.

In an embodiment, the hypermedia object comprises a button functionactivated by clicking the user interface input device when the cursor ispositioned within the associated responsive area, the cursor positioncontrolled by the two-dimensional pointing functions.

In an embodiment, the hypermedia object comprises a slider function.

In an embodiment, the hypermedia object comprises a menu function.

In an embodiment, the user input device is a computer mouse comprising afirst scrollwheel.

In an embodiment, the user input device is a computer mouse furthercomprising a second scrollwheel.

In an embodiment, the user input device is a computer mouse comprising atouchpad.

In an embodiment, the user input device is a computer mouse comprising aHigh Definition Touch Pad (HDTP).

In an embodiment, the user input device comprises a touch user interfaceresponsive to gestures and the at least one additional user-adjustableinput comprises at least one gesture.

In an embodiment, the user input device comprises a touch user interfaceresponsive to the yaw angle of a finger in contact with the touch userinterface and the at least one additional user-adjustable input isresponsive to a measurement of the yaw angle.

In an embodiment, the user input device comprises a touch user interfaceresponsive to the roll angle of a finger in contact with the touch userinterface and the at least one additional user-adjustable input isresponsive to a measurement of the roll angle.

In an embodiment, the user input device comprises a touch user interfaceresponsive to the pitch angle of a finger in contact with the touch userinterface and the at least one additional user-adjustable input isresponsive to a measurement of the pitch angle.

In an embodiment, the user input device comprises a touch user interfaceresponsive to at least two angles of a finger in contact with the touchuser interface and the at least one additional user-adjustable input isresponsive to a measurement of each of the two angles.

In an embodiment, a second displayed visual representation of thehypermedia object is displayed on the display screen when the hypermediaobject is activated.

In an embodiment, the first displayed visual representation of thehypermedia object changes when the hypermedia object is activated.

In an embodiment, the first displayed visual representation of thehypermedia object changes responsive to the at least one additionaluser-adjustable input.

In an embodiment, the user input device is a touch interface comprisinga tactile grammar.

Such extensions, features, and other aspects of the present inventionpermit far faster browsing, shopping, information gleaning through theenhanced features of these extended functionality roll-over andhyperlink objects.

Parameter Flow Routing to Multiparameter Hypermedia Objects

In an embodiment, an example general theme for extending traditional andcontemporary hypermedia objects such as hyperlink, roll-over, button,menu, and slider functions to handle the additional parameters providedby an APD include:

-   -   Test for cursor location meeting conditions for selecting the        MHO over other MHOs, background window, or user interface        modalities (for example, as was described earlier in conjunction        with FIG. 4);    -   If conditions met, additional user interface parameter signals        from APD are directed to software associated with the MHO.    -   The additional user interface parameter signals are used to        control aspects of the application or MHO that are not attained        by using user interface parameter signals with either the MHO or        with the traditional and contemporary forms of these hypermedia        objects.

In regards to this example general theme, it is noted that aside fromuse with scrollbar sliders and zoom sliders, the scrollwheel userinterface parameter signal of a contemporary scroll-wheel mouse do notappear to be used much (if at all) in the interaction with hypermediaobjects. Further, established use of the scrollwheel user interfaceparameter signal of a contemporary scroll-wheel mouse is fundamentaldifferent from the general theme of the invention: in existing systemsand software the scrollwheel user interface parameter signal providesthe same adjustment actions as would clicking on and dragging theslider, while in the example general theme this is precluded. Theinvention therefore provides for MHO hyperlinks, buttons, rollovers,menus, and sliders that direct scrollwheel user interface parametersignals to software associated with them (in keeping with the examplegeneral theme described above).

FIG. 29a depicts an arrangement provided for by the invention wherein anapplication designed to utilize additional APD user interface parameter,symbol, and/or gesture signals (thick arrowed lines) via MHOs comprisesat least one MHO (here a plurality are depicted, but only one isrequired). Traditional user interface parameter signals (thin arrowedline) are used to select (thick box lines) the application and aparticular MHO within it. The selected MHO permits the additional APDuser interface parameter, symbol, and/or gesture signals (thick arrowedlines) to be directed to the MHO associated software. The MHO associatedsoftware passes these on to the additional APD user interface parameter,symbol, and/or gesture signals (thick arrowed line) to the applicationsoftware. In an embodiment, the MHOs and application software canadditionally communicate information to the MHO associated software fromthe application software and/or from the MHO associated software to theapplication software (dashed arrowed lines).

In another embodiment, the selected MHO can further forward traditionaluser interface parameter signals as well as additional APD userinterface parameter, symbol, and/or gesture signals as suggested in FIG.29 b.

Implicit in the representative arrangements depicted in FIGS. 29a and29b is the process of focus selection within the context of theunderlying application. Such an arrangement can be implemented in anumber of ways, for example by the windowing and/or operating system(s),within the application, or various combinations of both. For example,the focus selection can be implemented within a hypermedia application(such as a browser or viewer), and in some embodiments further beassisted by the focus control functions within the windowing and/oroperating system(s). For example, FIG. 30a depicts a first examplearrangement for directing the user interface input parameter, symbol,and/or gesture signal stream to MHOs rendered within the display ofselected application, for example extending the signal flow depicted inFIG. 20. In this example, the focus control and focus selectionfunctions within the windowing and/or operating system(s) direct APDinformation flows to a particular selected application (here“Application K”), and within the selected application (“Application K”)a second focus selection function can be implemented. Note, for example,the selected application can be a browser-based application, as providedfor by the invention. As another example, FIG. 30b depicts a secondexample arrangement wherein the windowing and/or operating system(s)direct the user interface input parameter, symbol, and/or gesture signalstream directly to MHOs rendered within the display of selectedapplication.

Examples of MHO Structures

Like a traditional hypermedia object, an MHO can comprise one or morearguments including one or more of:

-   -   Graphics and image information;    -   Text string information;    -   Formatting information;    -   Hot-spot size and location details;    -   Modal behavior information (actions upon roll-over, actions upon        click-selection, etc.)    -   URL destination for a page or document location change.

Additionally, and MHO can also comprise:

-   -   Additional modal behavior information (extra states to be kept,        etc.)    -   A “name” (or link to) at least one instance of associated        software;    -   A list of variables used to communicate with associated        software;    -   An outgoing data interface to associated software;    -   An incoming data interface to associated software.

Examples of MHO Operation

The software associated with an MHO can use the extra parameters to do,for example, one or more of the following:

-   -   Change rendered graphics or images comprised by the MHO;    -   Change rendered graphics or images associated with but not        comprised by the MHO;    -   Change or produce sounds associated with the MHO;    -   Change software settings associated with the MHO;    -   Enter data to software associated with the MHO,        although the use and operation of the MHOs are not restricted to        these examples. The software associated with an MHO can perform        one or more functions such as these on its own (for example,        using information provided by arguments of the MHO function, as        in traditional HTML) or can be assisted or controlled in doing        so by the application software (employing for example one or        both of the communications paths represented by the dashed        arrowed lines).

Other Representative Types of MHOs

In addition to MHOs that are additional-parameter extensions oftraditional hypermedia objects, new types of MHOs unlike traditional orcontemporary hypermedia objects can be implemented leveraging theadditional user interface parameter signals and user interface metaphorsthat can be associated with them. Illustrative examples include:

-   -   Visual joystick (in various embodiments, the joystick could        retain its last position after it is “released,” or in other        embodiments can return to central position after release), for        example as depicted in FIG. 31;    -   Visual rocker-button (in various embodiments, the rocker button        can keep position after release, or in other embodiments can        return to central position after release), for example as        depicted in FIG. 32;    -   Visual rotating trackball, cube, or other object (in various        embodiments, the rocker button can keep position after release,        or in other embodiments can return to central position after        release), for example as depicted in FIG. 33;    -   A small miniature touchpad), for example as depicted in FIG. 34.

In any of these, the invention provides for the MHO to be activated orselected by various means, for example by clicking or tapping when thecursor is displayed within the area, simply having the cursor displayedin the area (i.e., without clicking or tapping, as in a rollover), etc.

Variations on any of these and as well as other new types of MHOs cansimilarly be crafted by those skilled in the art and these are providedfor by the invention.

MHO Dynamic Visual Features

In an embodiment, the displayed visual appearance of an MHO canadvantageously be changed response to at least one or more of thefollowing:

-   -   User interface parameter, symbol, and/or gesture signals        directed to the MHO and processed within the MHO;    -   User interface parameter, symbol, and/or gesture signals        directed to the MHO, passed to associated software, and        processed by the associated software to produce other signals,        data flows, or data changes which cause a change of displayed        visual appearance of the MHO;    -   Other actions of a associated software that produce other        signals, data flows, or data changes which cause a change of        displayed visual appearance of the MHO;        -   Internal operations of the associated software;        -   External data provided to the associated software;    -   External data provided to the MHO.        The invention also provides for the displayed visual appearance        of an MHO to be changed by other processes and circumstances.

In an embodiment, the aforementioned visual changes in the visualappearance of the MHO as displayed can comprise changes of displayedcolor in the visual display of the MHO. For example:

-   -   At least one color comprised in the visual display of the MHO        can vary as a user interface parameter is varied, directly        responsive to the value of the user interface parameter;    -   At least one color comprised in the visual display of the MHO        can vary as a user interface parameter is varied, responsive to        the action of associated software on the value of the user        interface parameter;    -   At least one color comprised in the visual display of the MHO        can vary as a user interface parameter is varied, responsive to        the action of associated software in response to other software        action on information responsive to the value of the user        interface parameter (for example, the return of a database        query, the calculation of a cost, etc.).

In an embodiment, the aforementioned visual changes in the visualappearance of the MHO as displayed can comprise changes of displayedtext in the visual display of the MHO. For example:

-   -   The content and/or format of text comprised in the visual        display of the MHO can vary as a user interface parameter is        varied, directly responsive to the value of the user interface        parameter;    -   The content and/or format of text comprised in the visual        display of the MHO can vary as a user interface parameter is        varied, responsive to the action of associated software on the        value of the user interface parameter;    -   The content and/or format of text comprised in the visual        display of the MHO can vary as a user interface parameter is        varied, responsive to the action of associated software in        response to other software action on information responsive to        the value of the user interface parameter (for example, the        return of a database query, the calculation of a cost, etc.).

In an embodiment, the aforementioned visual changes in the visualappearance of the MHO as displayed can comprise changes of displayedrendered graphics in the visual display of the MHO. For example:

-   -   At least one graphically rendered attribute comprised in the        visual display of the MHO can vary as a user interface parameter        is varied, directly responsive to the value of the user        interface parameter;    -   At least one graphically rendered attribute comprised in the        visual display of the MHO can vary as a user interface parameter        is varied, responsive to the action of associated software on        the value of the user interface parameter;    -   At least one graphically rendered attribute comprised in the        visual display of the MHO can vary as a user interface parameter        is varied, responsive to the action of associated software in        response to other software action on information responsive to        the value of the user interface parameter (for example, the        return of a database query, the rendering of a scene, etc.).

In an embodiment, the aforementioned visual changes in the visualappearance of the MHO as displayed can comprise changes of displayedimage in the visual display of the MHO. For example:

-   -   At least one displayed image comprised in the visual display of        the MHO can vary as a user interface parameter is varied,        directly responsive to the value of the user interface        parameter;    -   At least one displayed image comprised in the visual display of        the MHO can vary as a user interface parameter is varied,        responsive to the action of associated software on the value of        the user interface parameter;    -   At least one displayed image comprised in the visual display of        the MHO can vary as a user interface parameter is varied,        responsive to the action of associated software in response to        other software action on information responsive to the value of        the user interface parameter (for example, the return of a        database query, the display of a different view of an object or        scene, etc.).

Ranges and Cycles

In some applications, some visual changes in the visual appearance ofthe MHO as displayed will be rendered over a range with endpoints. Inother applications, some visual changes in the visual appearance of theMHO as displayed will be rendered over a periodic cycle withoutendpoints, in particular those relating directly or in metaphor to therotation of an object and/or viewpoint in one, two, or three angulardimensions.

FIG. 35 depicts a representative 1-dimensional array of N images, anyone of which can be selected for rendering as part or all of thedisplayed visual appearance of an MHO. In a first example, as a singleuser interface parameter is varied between values, one or another ofthese N images is displayed as part or all of the displayed visualappearance of an MHO, directly responsive to the last received value ofthe user interface parameter itself. In a second example, as a singleuser interface parameter is varied between values, one or another ofthese N images is displayed as part or all of the displayed visualappearance of an MHO, responsive to commands or data provided by anassociated program and/or other software.

FIG. 36 depicts a representative periodic structure imposed on the arrayof FIG. 35. Such a periodic structure is useful for example in renderingthe full 1-dimensional rotation of an object or viewpoint as well asother periodic behaviors or attributes (such as that of a clock face,weekly calendar, medicinal regiment, etc.).

FIG. 37 depicts a representative 2-dimensional array of N×M images, anyone of which can be selected for rendering as part or all of thedisplayed visual appearance of an MHO.

FIG. 38a depicts a representative 1-dimensional periodic structureimposed on the 2-dimensional array of FIG. 37. In this example, theremaining dimension of the array does not have a periodic structureimposed on it.

FIG. 38b depicts a representative 2-dimensional periodic structureimposed on the 2-dimensional array of FIG. 37.

FIG. 39 depicts a representative 2-dimensional array of K×M×N images,any one of which can be selected for rendering as part or all of thedisplayed visual appearance of an MHO.

FIG. 40 depicts a representative 1-dimensional periodic structureimposed on the 2-dimensional array of FIG. 39. In this example, theremaining dimension of the array does not have a periodic structureimposed on it. By substituting the 2-dimensional array structure in thedashed-line boxes with the FIG. 38a 1-dimensional periodic structureimposed on the 2-dimensional array, a resultant arrangement providing a2-dimensional periodic structure imposed on the 3-dimensional array ofFIG. 39 is obtained. By substituting the 2-dimensional array structurein the dashed-line boxes with the FIG. 38b 2-dimensional periodicstructure imposed on the 2-dimensional array, a resultant arrangementproviding a 3-dimensional periodic structure imposed on the3-dimensional array of FIG. 39 is obtained.

Although the above representative example structures have been describedin terms of images, the same structure can also be imposed on renderedgraphics instructions, text content, text format, and color data.

Example Applications Employing Multiparameter Hypermedia Objects

The following on-line and/or hypermedia catalog examples are intended todemonstrate advantages and value resulting from employing MHOs workingin conjunction with an APD so as to obtain more information far morequickly with far less effort. Some of these examples will showcasegeneral or specific use of the HDTP as the APD so as to demonstrate someof the powerful metaphors possible built from 6-dimensional positionsand movements of the human finger.

On-Line Shopping or Hypermedia-Catalog Examples

Traditional and contemporary webpage designs do not provide the freedomof easily altering the view presented for one section within a page(i.e., moving the view left or right, up or down, or zooming in or out)without affecting the display of the page itself. The following exampledemonstrates how one or more MHOs can be used to implement thisfunctionality.

FIG. 41 depicts a representative visual rendering of a body page 4101for a browser-rendered web-based application. In this example, the bodypage 4101 comprises a plurality of sections (here represented by4121-4125) providing product details for one or more selected categoriesfrom which the user may choose. In this example, various fields ofhypermedia or MHO hyperlinks 4111, 4112 are provided for navigating toother pages. Also in this example, a right-pointing hypermedia or MHObutton 4130 can be used for navigation to the next earlier-viewed page,for example the next category “Sale.” Additionally in this example,left-pointing hypermedia or MHO button 4140 can be used for navigationto previous earlier-viewed page, for example the “Spring/Summer 2009”page. Yet further in this example, a bottom-pointing hypermedia or MHObutton 4150 can be used for navigation to display additional sections.

One or more of these buttons 4130, 4140, 4150 (or other related buttonswhich can be included, such as “Next”, “Previous”, etc.) can beimplemented as MHOs. When such MHO buttons are selected or activated by(depending on the realization) being clicked, tapped or rolled over,additional parameters are directed to affairs associated with the button(for example, allowing a quick small overlay view to appear and bedisplay controlled (scroll up/down, scroll left/right, or zoom in/out)without disturbing the rest of the displayed webpage. For example, ifsuch an MHO button is (depending on the realization) click, tapped orrolled over, specific additional parameters (for example, fingermovements of roll, pitch, yaw, and/or downward pressure) can be directedto controlling the display (scroll up/down, scroll left/right, or zoomin/out) of the page associated with the MHO button. Another feature anMHO can be used to invoke is a quick small overlay view showingminiatures of previous earlier-viewed page(s) or next earlier-viewedpage(s).

In an example embodiment, the background of the body page can beconfigured as an MHO while background of each individual sections (hererepresented by 4121-4125) can be configured as an MHO separate from eachother and that of the MHO background of the body page. For example, ifthe area inside of body page 4101 is (depending on the realization)clicked, tapped, or rolled over, specific additional parameters (forexample, finger movements of roll, pitch, yaw, and/or downward pressure)can be directed to controlling the display (scroll up/down, scrollleft/right, or zoom in/out) of the body page 4101 itself. In anembodiment, tilting one's finger up and down in body page 4101 could beused to control the page's vertical movement (including section 4121).If instead a user (depending on the realization) clicks, taps or rollsover the area comprised by section 4121, the same control metaphor wouldinstead apply only to section 4121. Thus, in an embodiment wherein anHDTP is used to control such a MHO section area. If a user (depending onthe realization) taps or rolls over inside of section 4121 and tilts afinger up and down, only the display (scroll up/down, scroll left/right,or zoom in/out) of section 4121 will be affected, with no change to thedisplay (scroll up/down, scroll left/right, or zoom in/out) of the restof body page 4101.

Attention is now directed to controlling the display of sections andother features that facilitate rapid comparisons of products, services,images, feature options, items of information, etc. A motivation forthis is that most online shopping applications do not provide an easyway to compare products within the view of a webpage. Instead, onlineshoppers typically compare products by adding items of interest to theshopping cart or to a wish list, the collections of which are renderedas a separate page. Not only does the user have to switch between webpages in order to change, but such lists cannot easily be shared withfriends in such a way that maintains all images and metadata intact.Further, the shopping user has to have set up an account and login theaccount in order to view the wish list or shopping cart, adding yet moretime-consuming encumbrances to the on-line shopper, thus making theshopping experience more frustrating and time consuming and making thesite or catalog less attractive for return business. In an embodimentdescribed next, an alternative is provided in order to view and compareselected items one wishes. Aspects of the alternative can utilizevarious types of MHOs in various ways, for example as described in thematerial to follow.

FIG. 42 shows a group of sections, such as 4122, 4124-4127, hereoperating in different modality that those shown within the body page4101 of FIG. 41. To simplify FIG. 42, most other hypermedia objects orMHOs 4111, 4112, 4130, 4140 have been excised from the drawing.

Returning briefly to FIG. 41, sections 4121-4125 are currently displayedand additional sections can be displayed using the “More” hypermedia orMHO button 4150. Here, each section is separated by bright-shadeline-through bars 4101-4105. In an embodiment, when a line-through bar(such as one of 4101-4105) is double-clicked or double-tapped, anassociated section (for example the section above the line-through bar)is removed from the display and the line-through bar turns intodark-shade or black to signify that at least one section is hidden. Forexample in FIG. 42, sections for shoe styles A and C have been hiddenand their associated line-through bars 4101, 4103 are now converted intodark-shade or black bars 4201, 4203 to signify that at least oneimmediately preceding section is hidden. Other line-through bars (forexample 4102, 4104, 4105) are displayed unaffected. Notice in FIG. 42additional user operation and actions have resulted in displayed shoestyles B, D, E, F and G. Newly displayed sections that appear (such asshoe style F 4226 and shoe style G 4227 in FIG. 42) are, at leastinitially, displayed in conjunction with associated line-through bars(such as 4206 and 4207 in FIG. 42). In an embodiment, a dark-shade orblack bar 4201, 4203 can be double-clicked or double-tapped to revealthe concealed section(s) associated with it.

In an embodiment, the bright-shade line-through bar (such as one of4101-4105) or its equivalent can be implemented as a standard hyperlink.In an embodiment, the dark-shade or black bar (such as 4201, 4203) canalso be implemented as a standard hyperlink, or alternatively can beimplemented as an MHO hyperlink, rollover, button, etc. If implementedas an MHO, the extra parameters used to initiate and can be directed tocontrolling the display (scroll up/down, scroll left/right, or zoomin/out) of the concealed section(s) associated with it.

In an embodiment, an arrangement such as that of FIG. 42 can serve as apersonalized list. In an embodiment, an arrangement such as that of FIG.42 may not have resulted from a single body page such as the body page4101 of FIG. 41, but rather comprise or serve as a list that was createdby other means. In any of these cases or other approaches, the inventionprovides for one to additionally save the list for later use, save itand post on a blog or website, email the list to a friend or to oneself,or save the data to the computer or removable storage element (such as aremovable USB drive) for viewing at a later time when an active internetconnection is not available. The sections comprised by the list caninclude simple images or some of the more advanced image features to bedescribed next.

FIG. 43 provides a showcase view of a section of FIG. 41. Cursormovement within this section provides access to other hypermedla objectsor MHOs within the section which provide additional functionality. Forexample, should the section be larger than the displayed area 4121,additional user input parameters (for example, in the case of the HDTP,finger roll or pitch) could enact left/right or up/down scrolling of thematerial displayed within the section. As another example, rollovers orclicked/tapped buttons 4301-4304 can be used to change or modulate thecolor of the item displayed in the image field 4310 according toavailable product choices.

Positioning the cursor on the displayed in the image field 4310 can beused to invoke the display of an enlarged pop-up overlay image on thescreen. For example, using an HDTP touchscreen, the selection can bemade by placing the finger on the displayed image 4302. FIG. 44 shows anexample enlarged pop-up overlay image 4401 resulting from selecting adisplayed image field 4310 (such as the example shown in FIG. 43) afterchoosing a color from the options provided (for example via 4301-4304shown in FIG. 43).

In an embodiment, the enlarged pop-up overlay image 4401 is implementedas a MHO which provides the user not only a larger image of the depictedobject but also images corresponding to rotating the object through oneor more angles of rotation as can be provided by associated families ofimages. In an embodiment, the MHO provides the user with imagescorresponding to zooming in/out, changes in lighting, etc., as can beprovided by associated families of images and/or image processingcapabilities.

For example, employing the HDTP, finger position rotations of yaw,pitch, and roll, as suggested in the postures and gestures representedin FIG. 14e-14f , can be used to visually render a sequence of images ofthe object viewed from angularly-varied observation points, and use of asecond finger can be used to control the zoom by, for example, varyingthe spread between the two fingers as suggested in the postures andgestures depicted in FIG. 16. Further, for example, left-to-right motionof a thumb or little finger can be used to control the lighting effect.

In an embodiment, families of images visually render a sequence ofimages of the object viewed from angularly-varied observation points canbe implemented, for example, in a manner such as those described inconjunction with FIGS. 35-39. FIGS. 45a-45f depict representative viewsof a shoe as can be captured in individual images for use in therepresentative image array data structures described in conjunction withFIGS. 35-40. As an example of the user experience implying the HDTP:

-   -   FIG. 46a depicts a representative sequence of images capturing a        1-dimensional rotation of a shoe over a non-periodic range of        rotation angles as controlled by yaw positions and movements of        a single finger in contact with an HDTP;    -   FIG. 46b depicts a representative sequence of images capturing a        1-dimensional rotation of a shoe over a periodic range of        rotation angles as controlled by yaw positions and movements of        a single finger in contact with an HDTP;    -   FIG. 47 depicts a representative sequence of images capturing a        1-dimensional rotation of a shoe over a periodic range of        rotation angles as controlled by a family of two-finger posture        positions and movements finger in contact with an HDTP;    -   FIG. 48 depicts a representative sequence of images capturing a        1-dimensional rotation of a shoe within a different angle of        rotation that that of FIGS. 46a-46b over a range of rotation        angles as controlled by the pitch angle of a single finger in        contact with an HDTP.

In an embodiment, as the at least one additional user-adjustable inputfor entering values is varied between values, one or another image of agroup of images is displayed as part or all of the displayed visualappearance of an MHO, directly responsive to the last received value ofthe at least one additional user-adjustable input.

In an embodiment, zooming of the image can be implemented by well-knownimage processing algorithms and techniques which can be readily found intextbooks and on the internet. These are applied to one on morephotographic images associated with the currently active viewing angle.

In an embodiment, variation in lighting can be implemented by well-knownimage processing algorithms and techniques which can be readily found intextbooks and on the internet. A simple example is to varying theoverall brightness of the entire displayed image. In a moresophisticated embodiment, variation in lighting can be implemented, forexample, by interpolating between two or more photograph images takenunder different lighting conditions, said interpolation be implementedby well-known image processing algorithms and techniques which can bereadily found in textbooks and on the internet. A simple example isvarying the weighted proportions of each photograph image in a linearcombination of the component-by-component (RGB, YUV, HSB, etc.) pixelvalues. In yet another embodiment, a combination of proportional linearcombination of photograph image pixels values and other signalprocessing techniques can be used.

Many variations on the above examples are possible as is clear to oneskilled in the art. Overall, such innovations provide at least twoextraordinarily important advantages:

-   -   Rapid in-depth product inspection and comparison;    -   Significantly deeper understanding of the visual aspect of the        product;    -   Despite each of the above, far less wrist and hand fatigue in        the shopping experience.

Performance Seating Inspection Examples

FIG. 49 depicts an example ticket shopping webpage for a performancetheatre. When on-line shoppers seek to purchase tickets for concerts, acommon complaint is that the views provided of the venue (if any areprovided) are often confusing and misleading, and there is at bestlimited information provided regarding viewing distance and angles ofspecific sections to the stage and/or their proximity to the exits,pillars, and other aspects of the venue.

These shortcomings could be significantly addressed by displaying actualimages from the vantage point of each seat, ideally providing a 360°view from any given seat before purchase of (increasing expensive)tickets.

In an example embodiment, a potential ticket purchasing customer wouldcome to a ticket purchase website and after selecting a particular evenwould be presented with a web page such as that depicted in FIG. 49.Each section of seating can be, for example, signified with a differentcolor in order to distinguish different areas, types of areas, and/orprices. Seats that are not available could be signified in various ways,for example rendered in grey, black or other color, or displayed with an“X” or other symbol or highlighting rendered on (or replacing) the seaticon or depiction.

FIG. 50 depicts a popup image window overlay on the ticket shoppingwebpage depicted in FIG. 49. In an embodiment, the act of positioning ofthe cursor (or in an HDTP touchscreen or other touchscreenimplementation, positioning a finger) on one section (for example. Loge)an image of actual view would appear in an overly pop-up box. In anembodiment, this image shows a reasonably accurate rendering of the viewof the stage from that location, and in some embodiments the view ofother aspects of the theatre such as the aisles, exits, seat covers,etc. In an embodiment, the price of the ticket is also displayed. Bysimply sweeping the cursor (or finger on an HDTP touchscreen or othertouchscreen), the potential ticket purchaser is rapidly provided withmore accurate perception of what the viewing experience will be and itstrade-off with price, separation from friends, etc., as well as otherinformation relating to comfort, quality, crowding, exit access safety,etc. In an embodiment involving the HDTP and the pop-up box implementedas a MHO, the user can roll a finger left and right to view and decidewhat angle they prefer to have with their seats, or tilt a finger up ordown to move to a different row within the same section. For example,FIG. 51 depicts an overhead view showing a photographic representationof the separation distance from the stage. In an HDTP embodiment, roll,pitch, and/or yaw of the fingertip could allow viewing of the rest ofthe performing area including the ceilings, upper level(s), and exits.

In an example embodiment, tapping or clicking the graphic elementrepresenting a particular seat will select that seat as a further steptowards reservation and/or purchase.

In an embodiment, zooming of the image for a given separation distance(between the stage and a seating area actively under study) can beimplemented by known image processing algorithms and techniques whichcan be readily found in textbooks and on the internet. These are appliedto one on more photographic images associated with the currently activeviewing angle. In an embodiment, a mathematical model tied to a scaledseat-map or a database linked to a seat map (seat maps such as thatdepicted in FIG. 49) can calculate and/or retrieve separation distancedata and viewing angle data and present to one or both of an at leastone image selection element and an at least one image processingelement.

In an embodiment, an image selection element selects images to displaybased on calculated and/or retrieved viewing angle data. In anembodiment, an image selection element selects images to display basedon calculated and/or retrieved separation distance data. In anembodiment, should some locations in the venue contain viewobstructions, the image selection element can include provisions forselection specific image selection from obstruction-handling families ofimages.

In an embodiment, distance and/or angle information can be used by animage processing element to provide one or more of selective croppingand/or distance-varying image warping to render a reasonably accurateexpected view from the particular seat or region of seats. In anembodiment, at least some of the above can be used to provide display ofimages representing interactively selected views at various angles atthe particular seat or seating area. In an embodiment, these andadditional image processing functions can be used to implementpanoramically merged images.

Thus, the invention provides for

-   -   an improved interactive interface for consumers buying event        tickets online providing users with a virtual view of the event        venue as seen from any of the seats available for purchase.        Those who have never been to a given venue can experience the        view from a given seat to inform their selection before a        purchasing decision is made.    -   image processing to be used to synthesize an image of a        particular viewing angle from one or more photographic images        comprising one or more other viewing angle(s), at least one of        the calculation and display of which is under the control of a        user input device.    -   A mathematical model tied to a scaled seat-map or a database        linked to a seat map to be used to calculate and/or retrieve        separation distance data and viewing angle data and present to        one or both of an at least one image selection element and an at        least one image processing element, at least one of the        calculation and display of which is under the control of a user        input device.    -   Separation distance data and viewing angle data to be used by at        least one image processing element to calculate a synthesized        view from one or more photographic images, at least one of the        calculation and display of which is under the control of a user        input device.    -   An image selection element to be used to select images to        display based on calculated and/or retrieved viewing angle data.        In an embodiment, an image selection element selects images to        display based on calculated and/or retrieved separation distance        data.    -   Should some locations in the venue contain view obstructions,        the image selection element to include provisions for selection        specific image selection from obstruction-handling families of        images, at least one of the calculation and display of which is        under the control of a user input device.    -   Distance and/or angle information to be used by an image        processing element to provide one or more of selective cropping        and/or distance-varying image warping to render a reasonably        accurate expected view from the particular location, such as a        seat or region of seats in a theater, sports, or performance        venue, at least one of the calculation and display of which is        under the control of a user input device.    -   At least some of the afore described to be used to provide        display of images representing interactively selected views at        various angles at the particular seat or seating area, at least        one of the calculation and display of which is under the control        of a user input device.    -   At least some of the afore described and additional image        processing functions to be used to implement panoramically        merged images, at least one of the calculation and display of        which is under the control of a user input device.

While the invention has been described in detail with reference todisclosed embodiments, various modifications within the scope of theinvention will be apparent to those of ordinary skill in thistechnological field. It is to be appreciated that features describedwith respect to one embodiment typically can be applied to otherembodiments.

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentembodiments are therefore to be considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by the foregoingdescription, and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.

Although exemplary embodiments have been provided in detail, variouschanges, substitutions and alternations could be made thereto withoutdeparting from spirit and scope of the disclosed subject matter asdefined by the appended claims. Variations described for exemplaryembodiments may be realized in any combination desirable for eachparticular application. Thus particular limitations, and/or embodimentenhancements described herein, which may have particular advantages to aparticular application, need not be used for all applications. Also, notall limitations need be implemented in methods, systems, and apparatusesincluding one or more concepts described with relation to the providedexemplary embodiments. Therefore, the invention properly is to beconstrued with reference to the claims.

What is claimed is:
 1. A system for activating a hypermedia objectassociated with an application, the system comprising: a display havinga display screen; a user interface input device configured to providetwo-dimensional pointing functions and at least one additionaluser-adjustable input configured to input a value from a rangecomprising more than two possible values; and at least one processor,wherein the at least one processor is configured to: activate thehypermedia object associated with the application; display thehypermedia object on the display screen; allow a user to activate thehypermedia object from a user-initiated action enacted on the userinterface input device, wherein the hypermedia object comprises: a firstvisual representation of the hypermedia object for display in a firstregion of the display screen, wherein the first region of the displayscreen is associated with the application, and an associated responsivearea in a second region of the display screen for activating thehypermedia object, and wherein the hypermedia object is responsive toinformation provided by the user interface input device, whereinactivating the hypermedia object enables entry of at least oneadditional user-adjustable input value for use by the application usingthe user interface input device, wherein the at least one additionaluser-adjustable input value comprises a finger angle based on a measuredparameter of a single contact location of a finger on the user interfaceinput device.
 2. The system of claim 1, wherein the first and secondregions of the display are the same region.
 3. The system of claim 1,wherein the user interface input device is further configured toactivate a hyperlink function when a cursor is positioned within-theassociated responsive area, wherein a cursor position is controlled bythe two-dimensional pointing functions of the user interface inputdevice.
 4. The system of claim 1, wherein the hypermedia objectcomprises a rollover function that is activated by using the userinterface input device to position a cursor within the associatedresponsive area, wherein a cursor position is controlled by thetwo-dimensional pointing functions of the user interface input device.5. The system of claim 1, wherein the hypermedia object comprises abutton function that is activated by the user interface input devicewhen a cursor is positioned within the associated responsive area,wherein a cursor position is controlled by the two-dimensional pointingfunctions of the user interface input device.
 6. The system of claim 1,wherein the hypermedia object comprises a slider function.
 7. The systemof claim 1, wherein the hypermedia object comprises a menu function. 8.The system of claim 1, wherein the user interface input device is acomputer mouse comprising a first scrollwheel.
 9. The system of claim 8,wherein the user interface input device is the computer mouse furthercomprising a second scrollwheel.
 10. The system of claim 1, wherein theuser interface input device is a computer mouse comprising a touchpad.11. The system of claim 1, wherein the user interface input device is acomputer mouse comprising a High Definition Touch Pad (HDTP).
 12. Thesystem of claim 1, wherein the user interface input device comprises atouch user interface responsive to gestures, and the at least oneadditional user-adjustable input comprises at least one gesture.
 13. Thesystem of claim 1, wherein the user interface input device comprises atouch user interface responsive to a yaw angle of the finger in contactwith the touch user interface, and the at least one additionaluser-adjustable input is responsive to a measurement of the yaw angle.14. The system of claim 1, wherein the user interface input devicecomprises a touch user interface responsive to a roll angle of thefinger in contact with the touch user interface, and the at least oneadditional user-adjustable input is responsive to a measurement of theroll angle.
 15. The system of claim 1, wherein the user interface inputdevice comprises a touch user interface responsive to a pitch angle ofthe finger in contact with the touch user interface, and the at leastone additional user-adjustable input is responsive to a measurement ofthe pitch angle.
 16. The system of claim 1, wherein the interface userinput device comprises a touch user interface responsive to at least twoangles of the finger in contact with the touch user interface, and theat least one additional user-adjustable input is responsive to ameasurement of each of the at least two angles.
 17. The system of claim1, wherein a second visual representation of the hypermedia object isdisplayed on the display screen responsive to control of the hypermediaobject.
 18. The system of claim 1, wherein the first visualrepresentation of the hypermedia object changes responsive to control ofthe hypermedia object.
 19. The system of claim 1, wherein the firstvisual representation of the hypermedia object changes responsive to theat least one additional user-adjustable input.
 20. The system of claim19, wherein a second visual representation of the hypermedia object isdisplayed responsive to control of the hypermedia object.
 21. The systemof claim 1, wherein the user interface input device is a touch interfacecomprising a tactile grammar.
 22. The system of claim 1, wherein thefinger angle is based on at least one measured parameters of at leastone of roll, pitch, or yaw of the finger on the user interface inputdevice.
 23. The system of claim 1, wherein the finger angle includes aroll, pitch, and yaw of the finger on the user interface input device.24. The system of claim 1, wherein the finger angle is based on at leastone measured parameter with shape recognition.
 25. The system of claim1, wherein the finger angle is based on at least one measured tilt ofthe finger on the user interface input device.
 26. The system of claim25, wherein the finger angle is based on at least one measured rotationof the finger on the user interface input device.
 27. The system ofclaim 1, wherein an additional user-adjustable input for entering valuesis within a hotspot.
 28. The system of claim 1, wherein activating thehypermedia object by the user interface input device comprises pointingwith a pointing device or entering a keystroke.